'When We Find Something That Doesn’t Work Well Enough, We Replace It in Order to Develop a More Effective Approach'
The winners of the third annual Ilya Segalovich Award were recently announced in Moscow. Established by Yandex, the award promotes the scientific endeavours of young researchers from Russia, Belarus, and Kazakhstan in the field of Computer Science. Among this year’s winners were three HSE students, including Alexander Grishin, a Doctoral student of the Big Data and Information Retrieval School of the HSE Faculty of Computer Science. Alexander spoke to us about his work, research challenges, and why he was surprised to receive the award.
HSE University and My Research Work
I started doing research work as an undergraduate in the Faculty of Biological and Medical Physics at the Moscow Institute of Physics and Technology (MIPT). In my third and fourth years there, I used machine learning to analyse DNA nucleotide sequences. It wasn’t an easy task—I had never studied machine learning before and I didn’t have enough knowledge of biology. I was keen on the subject, but I felt I lacked hard skills.
I realized that I was more interested in the technical side of things than the biology side, and I started looking for a place where I could gain those skills. That’s how I ended up in HSE University’s ‘Data Science’ Master’s Programme at the School of Data Analysis. The first year required a lot of intensive study, so I had no time for research. Then I met Professor Dmitry Vetrov, and we’ve been doing research together ever since.
My work on Professor Vetrov’s research team started with the biology side of things. I joined Insilico Medicine, a company that synthesizes pharmaceuticals. Alongside my academic advisor Daniil Polykovsky, I worked on forecasting the properties of molecules. This would enable us to generate molecules with the same properties for medicinal purposes.
Solving a difficult task requires solving easier ones first, so we started by trying to determine whether or not a molecule had any specific properties. We never published the method we developed, as two very similar articles were released around that time and the authors of those articles used a slightly better approach. That’s when we knew that our work was sound and up-to-date, just not quick enough.
After that, I began looking for another area of research. I wanted to work with Pavel Shvechikov and luckily he was able to take me on. I wanted to explore a new field, and that ended up being reinforcement learning. Pavel and I worked together for a few years. Our team gave a presentation at the NeurIPS oral workshop and wrote an article for ICML.
Our team doesn’t specialize in reinforcement learning, so we sometimes lack the level of expertise we’d like. But this is common, as not many people work on reinforcement learning at a serious academic level. We decided to put together a team dedicated to the field. As for the commercial sector, companies have always been interested in this area, but it remains underdeveloped.
The Prize-Winning Research
The things we’re doing are quite simple. Rather than focus on specific tasks, we develop general algorithms. We take an academic approach and try to develop more effective algorithms that can cover a wide range of tasks. We also study existing algorithms and the way they work in order to uncover potential problems. To put it simply, we analyze each individual element and when we find something that doesn’t work well enough, we replace it in order to develop a more effective approach.
We received the Ilya Segalovich Award for our article about overestimation bias in reinforcement learning. After performing an action, we try to figure out whether these actions will have good or bad consequences. Previously, we have relied on very primitive tools such as understatement or artificial suppression to reduce overestimation bias. But we’ve developed a more flexible and effective tool to avoid this kind of overestimation. Samsung has recognized our research too, which was a surprise to me. I thought the company would be more interested in practical results, while our work has the academic focus of enhancing the effectiveness of reinforcement learning.
Are there traditional tasks in reinforcement learning? Yes, and they’re used to test algorithms and compare them. One example is Atari games, in which the agent sees a picture, another image of the game, and performs a certain action (controlling the agent). These games vary both visually and mechanically.
We tested our algorithms on ‘locomotor tasks’—a set of tasks in a physical simulator that models different kinds of robots, from simple robots with a couple of joints to complex humanoid ones. The agent receives signals from sensors informing it of the position, angle, and speed of each limb. Every 15 milliseconds, the agent must exert a force to change the position of the body. Then the agent must exert more force to perform a new task, such as running as fast as it can without falling.
Reinforcement learning involves working with states—things the agent sees or does depending on the state it finds itself in, or rewards it receives for performing the right actions or sequences. The great thing about it is that the specifics of these states—where they come from, what they look like, how they change—don’t really matter. This is because learning algorithms are universal and the task is always the same: to teach the agent the best approach that produces the biggest reward. From a theoretical standpoint, the reward itself isn’t important, be it the speed of the robot’s movements, efficient power consumption, a click from a user, or money made at the stock exchange. The methods we used to teach robots how to run (or rather, to learn how to run themselves) could be used to distribute power generated by a nuclear power plant or advertise goods to consumers. Reinforcement learning has limitless applications.
The Yandex Prize and My Hobby
I’m delighted we received the award for a number of reasons. Firstly, personal motivation and external recognition often don’t go hand-in-hand in academia. Everyone has had their articles turned down more than once. So an award is something extraordinary, and it drives you to do better. Another important aspect is that the winners of this award receive substantial financial support, access to Yandex Toloka, and an invite to a conference. These are great motivators too.
Secondly, I didn’t think I had much of a chance of winning because I’ve had very few things published, so the award came as a pleasant surprise. The organizers must have seen me as a researcher capable of creating something interesting and useful. The number of publications wasn’t a deciding factor in their decision.
I was also delighted to tell Professor Vetrov (he wasn’t eligible for the award, although I’m sure he would have won if he had been) and my friends. Even if they don’t know much about science or research, they think it’s cool and they’re happy for me.
The award ceremony took place almost at the same time as the first concert of the ensemble I’m in, which is made up of six cellists and a pianist. We’ve got quite a wide repertoire, so there’s bound to be something for everyone to enjoy. We play everything from Shostakovich waltzes to the Game of Thrones theme. That’s another achievement I’m proud of.
Alaa N. Assaf is a doctoral student at HSE University’s Faculty of Law. In addition to doing research for his PhD thesis, he teaches undergraduates. Alaa has shared his thoughts on studying international law, life in Moscow, and what he misses about teaching in person.
HSE University continues to develop its new study format for students embarking on a research career: the Combined Master's-PhD track. This year, there will be 75 places for Master’s students on the track—two thirds more than last year. HSE Vice Rector Sergey Roshchin talks about the appeal of the combined-track option, how to enrol, and the achievements of last year’s applicants.
The British Educational Research Association, BERA, has announced the winners of its Early Career Researchers Award, founded this year to recognize emerging researchers. The new award’s first two winners are Saule Bekova, Research Fellow at theInstitute of Education of the Centre of Sociology of Higher Education, and Binwei Lu of Durham University (UK). In the announcement, the organizing committee praised them as ‘outstanding scholars’ and noted the potential significance of their research for the field of education.
At the 2021 RusCrypto Competition, HSE PhD student Anastasia Malashina (MIEM) won the student report category by the unanimous decision of the competition committee, which included the directors of the RusCrypto-2021 Association, members of the programme committee, and experts in the field of computer security. Anastasia spoke with HSE News Service about her presentation.
In 2010, HSE University launched the Full-time Advanced Doctoral Programme, an innovative programme for doctoral students at that time. At the beginning of 2021, an alumna of the programme’s first cohort became the first programme graduate to earn her Doctor of Sciences degree: Elena Vakulenko, Assistant Professor at HSE University’s Faculty of Economic Sciences. In an interview with HSE News Service, the professor discussed her academic career.
‘We Facilitate High-Speed Car Crashes and Study How Car Engines Work Based on Photos of Flying Debris’
Nikita Kazeev holds a Candidate of Sciences degree (Russian equivalent of a PhD) in Computer Science and a PhD in Physics. He is a Research Fellow at the LAMBDA Laboratory and works at CERN. In an interview with HSE News Service, he talked about what it was like to defend his dissertation in a double doctoral degree programme at HSE University and Sapienza University of Rome, what it is like to conduct research in Geneva, and why it is imperative to communicate with colleagues.
In Pursuit of the African Migrant Experience: HSE Doctoral Student Sheds Light on an Understudied Segment of the Moscow Population
At the recent seminar of the HSE International Laboratory for Social Integration Research, Isaac Olumayowa Oni, HSE doctoral student and lab research assistant, spoke about African migration to Russia. In an interview with HSE News Service, Isaac discussed his research and talked about what it’s like to study at HSE and live in Moscow.
More than 56,000 international participants from 197 countries have registered for Open Doors: Russian Scholarship Project organized by Association of Global Universities for master’s and doctoral programme applicants.Registration for the competition closed on December 10, 2020.
Zaruhi Hakobyan earned her Bachelor’s degree in Yerevan, her Master’s from HSE University, and her PhD from both HSE University and the University of Luxembourg. She spoke to the HSE News Service about the differences in academic approaches at various universities.
Vladislav Khvostov, a second-year student of the Doctoral School of Psychology at HSE University, was recognized with a Graduate Conference Award for Student Members from the Psychonomic Society. Every year a total of 20 graduate students in cognitive psychology receive this prestigious international award in psychology. HSE News Service spoke with Vladislav about the award, his interest in cognitive psychology and research.