• A
  • A
  • A
  • ABC
  • ABC
  • ABC
  • А
  • А
  • А
  • А
  • А
Regular version of the site
  • HSE University
  • News
  • Russian Scientists Teach AI to Analyse Emotions of Participants at Online Events

Russian Scientists Teach AI to Analyse Emotions of Participants at Online Events

Russian Scientists Teach AI to Analyse Emotions of Participants at Online Events

© iStock

HSE researchers have proposed a new neural network method for recognising emotions and people's engagement. The algorithms are based on the analysis of video images of faces and significantly outperform existing single models. The developed models are suitable for low-performance equipment, including mobile devices. The results can be implemented into video conferencing tools and online learning systems to analyse the engagement and emotions of participants. The results of the study were published in IEEE Transactions on Affective Computing.

The COVID-19 pandemic resulted in the active development of online video conferencing tools and e-learning systems. Artificial intelligence technologies can help teachers remotely monitor the engagement of event participants. Currently, algorithms for analysing student behaviour and identifying engagement in the online environment are being studied by experts in the field of data mining for education. Automatic methods based on computer vision technologies are the most widespread analysis tools. The researchers believe that the quality of many e-learning systems can be greatly influenced by identifying the emotions and involvement of participants based on video analytics.

HSE researchers have developed a new neural network algorithm for recognising emotions and engagement from video images of faces as part of the project “Neural network algorithms for analysing the dynamics of students' emotional state and engagement based on video imaging data” of the HSE AI Centre.

The researchers taught the neural network to extract emotional features based on a specially developed robust optimisation technique and process only the most important areas of the face. First, the neural network detects faces and extracts their emotional features, then groups the faces of each participant. Then, with the help of specially trained effective neural network models, the emotional features of each selected person are extracted, aggregated using statistical functions, and classified. At the final stage, the fragments of the video tutorial with the most pronounced emotions and different degrees of involvement of each listener are visualised. As a result, the researchers managed to create a new model that determines the emotions of each person and their engagement for several faces in the video at once.

Andrey Savchenko

Andrey Savchenko, Project Head, Professor at the Department of Information Systems and Technologies

‘We have demonstrated that the proposed algorithms outperform existing single models in terms of accuracy for several datasets. At the same time, unlike most well-known technologies, the developed models can engage in real-time video processing even on low-performance devices, including the mobile phones of each participant of an online event. Together with Ilya Makarov from the Artificial Intelligence Research Institute, we have created an easy-to-use computer programme that allows you to process a video recording of a webinar or online class and get a set of video clips with the emotional features of each participant.’

The results can be implemented into video conferencing and online learning software to analyse the engagement and emotions of participants. Thus, during the preliminary testing of the online course, we can use students’ reactions to understand which parts of the lecture were most interesting, and what turned out to be difficult to understand and needs to be corrected. Currently, the researchers are working to integrate the developed models into the Jazz by Sber video conferencing service. The videos collected from open sources for this project will allow researchers to take a step towards creating a service for identifying emotions and engaging students in online events.

See also:

‘In the Future, I Expect Rapid Development of Professions Related to Prompt Engineering’

The English-language programme of HSE Online ‘Master of Computer Vision’ will change its name to ‘Artificial Intelligence and Computer Vision’ in 2024. Andrey Savchenko, the programme academic supervisor, shares how the new name will affect the programme semantics, why AI has become the main federal trend in the field of information technology, and what tasks graduates will solve.

Artificial Intelligence as a Driver of Digital Transformation

In December, the HSE Institute for Statistical Studies and Economics of Knowledge and the HSE AI Research Centre participated in UNCTAD eWeek to discuss the future of the emerging digital economy. One of the topics discussed during the conference was artificial intelligence and its applications in driving the digital transformation of industry sectors. The session was co-organised by HSE University.

HSE University Receives Highest Grant under Priority 2030 Programme

HSE University has proved its leading position in the first group of the ‘Research Leadership’ field under the Priority 2030 programme. The university has also received the highest grant for teaching digital competencies to students, demonstrating its educational leadership in the fields of digital technologies and AI.

‘The Future Lies with AI Technologies and HSE University Understands That’

At the AI Journey 2023 international conference in Moscow, a ranking of Russian universities that train the best AI specialists was published. HSE University entered the A+ leadership group, taking first place according to such criteria as ‘Demand for hiring graduates’, ‘Quality of educational environment’, and ‘Activities for the development of school education’. Ivan Arzhantsev, Dean of HSE University’s Faculty of Computer Science, spoke to the HSE News Service about how AI specialists are trained at HSE University and what plans the university has in this area.

‘Every Article on NeurIPS Is Considered a Significant Result’

Staff members of the HSE Faculty of Computer Science will present 12 of their works at the 37th Conference and Workshop on Neural Information Processing Systems (NeurIPS), one of the most significant events in the field of artificial intelligence and machine learning. This year it will be held on December 10–16 in New Orleans (USA).

Specialists from the HSE Institute of Education Confirm GigaChat’s Erudition in Social Sciences

A multimodal neural network model by Sber, under the supervision of HSE University’s expert commission, has successfully passed the Unified State Exam in social studies. GigaChat completed all exam tasks and scored 67 points.

HSE University Students Win in the AIJ Science Competition at AI Journey 2023

The International Sber Conference of Artificial Intelligence, ‘AI Journey 2023’ recently took place in Moscow. Alexander Rogachev, doctoral student of the HSE Faculty of Computer Science, and Egor Egorov, an HSE 4th-year undergraduate student became the winners of the AIJ Science competition for scientific articles on artificial intelligence that was held as part of the event. The research was carried out under the umbrella of the HSE's Laboratory of Methods for Big Data Analysis (LAMBDA).

HSE University Hosts Fall into ML 2023 Conference on Machine Learning

Over three days, more than 300 conference participants attended workshops, seminars, sections and a poster session. During panel discussions, experts deliberated on the regulation of artificial intelligence (AI) technologies and considered collaborative initiatives between academic institutions and industry to advance AI development through megaprojects.

Child Ex Machina: What Artificial Intelligence Can Learn from Toddlers

Top development teams around the world are trying to create a neural network similar to a curious but bored three-year-old kid. IQ.HSE shares why this approach is necessary and how such methods can bring us closer to creating strong artificial intelligence.

‘My Research Has Evolved into A Broader and More Encompassing Vision’

Seungmin Jin, from South Korea, is researching the field of Explainable AI and planning to defend his PhD on ‘A Visual Analytics System for Explaining and Improving Attention-Based Traffic Forecasting Models’ at HSE University this year. In September, he passed the pre-defence procedure at the HSE Faculty of Computer Science School of Data Analysis and Artificial Intelligence. In his interview for the HSE News Service, he talks about his academic path and plans for the future.