‘Speech, Facial Expressions, and Gestures Cannot Lie’

© HSE University

Would you like to know whether a speaker’s trembling voice or an accidental gesture can give them away? At HSE University in Nizhny Novgorod, researchers are developing an algorithm that analyses speech, facial expressions, and gestures, and determines whether information is truthful with 92% accuracy. The project has applications ranging from forensic examination and bank recruitment to fundamental research. Anna Khomenko, head of the research group and Senior Research Fellow at the Centre for Language and Brain at the HSE Faculty of Humanities in Nizhny Novgorod, explains how students and researchers are working together to create a corpus of video recordings, train a classifier, and prepare to introduce computer vision technology.

This article continues the Schola series on the projects of HSE University’s new research and study groups.

— How did the idea for the project come about?

— The project, ‘Speech, Facial Expressions, and Gestures Cannot Lie: Assessing the Veracity of Information Using Psycholinguistic Methods,’ develops last year’s project, ‘Truth or Lie? Determining the Reliability/Unreliability of Information from Multimodal Text.’ The original project was inspired by Eva Patrakova, a student on the Fundamental and Applied Linguistics programme at HSE University in Nizhny Novgorod. She worked on methods for forensic psychological assessment for Russian legal linguistics and concluded that there was a gap in this field. The members of our research and study group therefore set out to address it through their research.

In 2025, based on previous Russian and international studies, our team developed a model for analysing the truthfulness of statements and tested it on a limited sample. The model achieves 92% accuracy on the test material. We are currently in the process of registering intellectual property rights for the model we have developed.

— What are the objectives of your new project?

— This year, we are refining the algorithm, testing its accuracy on a larger volume of data, and developing an interface. The main goal of this year’s project is to create an automated model—and, in the longer term, a fully automatic model using computer vision—for recognising truthful and untruthful information based on a speaker’s speech, facial expressions, and gestures. To create such a model, over the course of the year we need to compile a corpus of video recordings containing truthful and false information with a total duration of at least 1,000 minutes. We then need to annotate the speakers’ speech, facial expressions, and gestures in accordance with a discourse annotation protocol; calculate statistically significant speech, facial, and kinesic characteristics; further train the classifier on this sample; and create an interface for the algorithm that can be used by external users. At present, we are already further training the classifier and are ready to move on to creating an interface for our model.

— Why is the project important?

— There is demand for the development of such an algorithm in several fields, both fundamental and practice-oriented.

For fundamental research, it is important to study Russian-language discourse involving the production of false information in spoken communication. Corpora of written texts containing false statements already exist for Russian, but an oral corpus has yet to be developed.

From a practical perspective, our research could be applied both in forensic psychological and linguistic assessment and in pre-trial investigations, for example, when analysing testimony. Law enforcement agencies have already expressed interest in the development. Psychologists and psychiatrists are also showing considerable interest, as the technology could provide a basis for screening diagnostics, including for people with addictions.

Commercial organisations, particularly large banking ecosystems, have expressed interest in the research, with a view to using it in recruitment procedures, for example during interviews with prospective employees.

However, it is important to understand that, at the current stage of scientific, technological, and legal development in Russia, no algorithm in any of these fields makes the final decision as to whether a person is lying or telling the truth. Any automatic or automated system is merely a tool designed to help a human see and analyse the material.

— What results do you expect to achieve?

— In addition to an interface-based algorithm for determining the truthfulness or untruthfulness of information from speech, facial expressions, and gestures, an important outcome of our research is a labelled corpus of multimodal texts. This corpus makes it possible to identify language-specific characteristics of how false and truthful information is produced—characteristics associated with the Russian language and the mentality of its speakers—as well as markers of the truthfulness or falsity of statements in Russian speech. Such knowledge about Russian-language discourse can be considered fundamental.

At the next stage, in 2027, we hope to receive grants from commercial partners and government funds to integrate computer vision systems into our model, followed by the full automation of the screening algorithm.

— Your research and study group is called CompLing v.2. What does the name stand for? Could you tell us more about it?

— CompLing is an abbreviation of computer linguists, while v.2 indicates the second iteration of our work as part of the research group. The name was created after the model of the group called CorpLings (corpus linguists), which was established in 2012 by the distinguished scholars Olga Lyashevskaya and Vera Sibirtseva. We can say that the CompLings continue the traditions of a research school founded by a group of scholars from Moscow and Nizhny Novgorod more than a decade ago.

— What challenges have you encountered or do you expect to encounter?

— Research into spoken language, particularly when several modalities are involved—speech, as well as facial expressions and gestures captured in video recordings—presents challenges related to collecting and processing the material. First, we have to address issues concerning the storage and processing of personal and biometric data. Second, we have to overcome the difficulties of operationalising the analysis of multi-stream data, including developing protocols and automated systems. All these problems can be addressed within the existing legal and scientific framework. HSE’s administrative and research support services are actively helping us to tackle them, and we are extremely grateful to them for that!

— How were the project participants selected? Could you tell us about the key members of the team?

— The selection of project participants was based on two main principles: having the necessary competencies and being able to work as part of a team.

Eva Patrakova and I are developing methods for analysing speech, gesture, and facial-expression data. Polina Zhilina is developing methods for the automatic transcription of spoken language. Mariya Khudyakova and Artem Mikulinsky are working on protocols for annotating the material for the corpus. Ruslana Zobnina, Anna Vorontsova, and Natalia Efimenko are annotating the corpus material. Anastasiia Komratova is turning everything we have created into a computer programme and developing classification algorithms.

Quite a large number of undergraduate and master’s students are involved in our project. For them, the research and study group is an opportunity to work with more experienced colleagues and learn from their expertise and knowledge. But that is not the only benefit. The technologies we are developing are also based on the skills that each participant brings to the project from the outset. Polina Zhilina, for example, had already spent several years working on automatic speech-recognition models before joining the project, and her expertise prompted us to improve these models for our specific tasks. HSE students have a very broad range of competencies, acquired both through their academic courses and through work placements and professional activities—many of them are already working in their field while still studying. This means that, when tackling real-world professional tasks, we can rely on them as specialists. Everyone has their own responsibilities corresponding to their particular competencies; everyone has an important role to play, and the project would not have happened without any one of us.

Of course, we are all actively writing academic papers and taking part in conferences to raise awareness of our work. The most recent and possibly the most important conference for which we prepared material was ‘Digitalisation of Industrial Russia,’ held from May 18 to 21. We hope that this event will help us find new partners.

— What are the future applications of your project?

— Our research has both academic and commercial potential.

As far as fundamental science is concerned, all new theories, hypotheses, and developments are built on existing knowledge. What is important here is that, on the one hand, early-career researchers of the groups become familiar with the existing theoretical foundations, while on the other, they contribute to creating new ones. After all, to develop a new scientific product, it is first necessary to lay its fundamental foundations.

The research we have already carried out as part of the project feeds directly into academic courses in psycholinguistics, communication studies, and related fields. It is also actively used by young project participants in their graduation theses and PhD dissertations.

In addition, all the research and study group projects provide an excellent starting point for the academic careers of young participants—both students and postgraduate students. They learn how to communicate within the academic community, give presentations at research seminars, write abstracts and papers, and participate in conferences. The research and study groups provide financial support and incentives for these activities. They are an excellent tool for immersing young people in research through genuinely interesting projects with significant social value. As for the commercialisation of our research, we are currently actively negotiating with partners from the public and private sectors to develop customised algorithms for determining the truthfulness or untruthfulness of information based on speech, facial expressions, and gestures and tailored to specific client needs. We are also open to dialogue with new academic and commercial partners to develop further solutions in related fields.

See the original text in Russian on Schola.