'When We Find Something That Doesn’t Work Well Enough, We Replace It in Order to Develop a More Effective Approach'
The winners of the third annual Ilya Segalovich Award were recently announced in Moscow. Established by Yandex, the award promotes the scientific endeavours of young researchers from Russia, Belarus, and Kazakhstan in the field of Computer Science. Among this year’s winners were three HSE students, including Alexander Grishin, a Doctoral student of the Big Data and Information Retrieval School of the HSE Faculty of Computer Science. Alexander spoke to us about his work, research challenges, and why he was surprised to receive the award.
HSE University and My Research Work
I started doing research work as an undergraduate in the Faculty of Biological and Medical Physics at the Moscow Institute of Physics and Technology (MIPT). In my third and fourth years there, I used machine learning to analyse DNA nucleotide sequences. It wasn’t an easy task—I had never studied machine learning before and I didn’t have enough knowledge of biology. I was keen on the subject, but I felt I lacked hard skills.
I realized that I was more interested in the technical side of things than the biology side, and I started looking for a place where I could gain those skills. That’s how I ended up in HSE University’s ‘Data Science’ Master’s Programme at the School of Data Analysis. The first year required a lot of intensive study, so I had no time for research. Then I met Professor Dmitry Vetrov, and we’ve been doing research together ever since.
My work on Professor Vetrov’s research team started with the biology side of things. I joined Insilico Medicine, a company that synthesizes pharmaceuticals. Alongside my academic advisor Daniil Polykovsky, I worked on forecasting the properties of molecules. This would enable us to generate molecules with the same properties for medicinal purposes.
Solving a difficult task requires solving easier ones first, so we started by trying to determine whether or not a molecule had any specific properties. We never published the method we developed, as two very similar articles were released around that time and the authors of those articles used a slightly better approach. That’s when we knew that our work was sound and up-to-date, just not quick enough.
After that, I began looking for another area of research. I wanted to work with Pavel Shvechikov and luckily he was able to take me on. I wanted to explore a new field, and that ended up being reinforcement learning. Pavel and I worked together for a few years. Our team gave a presentation at the NeurIPS oral workshop and wrote an article for ICML.
Our team doesn’t specialize in reinforcement learning, so we sometimes lack the level of expertise we’d like. But this is common, as not many people work on reinforcement learning at a serious academic level. We decided to put together a team dedicated to the field. As for the commercial sector, companies have always been interested in this area, but it remains underdeveloped.
The Prize-Winning Research
The things we’re doing are quite simple. Rather than focus on specific tasks, we develop general algorithms. We take an academic approach and try to develop more effective algorithms that can cover a wide range of tasks. We also study existing algorithms and the way they work in order to uncover potential problems. To put it simply, we analyze each individual element and when we find something that doesn’t work well enough, we replace it in order to develop a more effective approach.
We received the Ilya Segalovich Award for our article about overestimation bias in reinforcement learning. After performing an action, we try to figure out whether these actions will have good or bad consequences. Previously, we have relied on very primitive tools such as understatement or artificial suppression to reduce overestimation bias. But we’ve developed a more flexible and effective tool to avoid this kind of overestimation. Samsung has recognized our research too, which was a surprise to me. I thought the company would be more interested in practical results, while our work has the academic focus of enhancing the effectiveness of reinforcement learning.
Are there traditional tasks in reinforcement learning? Yes, and they’re used to test algorithms and compare them. One example is Atari games, in which the agent sees a picture, another image of the game, and performs a certain action (controlling the agent). These games vary both visually and mechanically.
We tested our algorithms on ‘locomotor tasks’—a set of tasks in a physical simulator that models different kinds of robots, from simple robots with a couple of joints to complex humanoid ones. The agent receives signals from sensors informing it of the position, angle, and speed of each limb. Every 15 milliseconds, the agent must exert a force to change the position of the body. Then the agent must exert more force to perform a new task, such as running as fast as it can without falling.
Reinforcement learning involves working with states—things the agent sees or does depending on the state it finds itself in, or rewards it receives for performing the right actions or sequences. The great thing about it is that the specifics of these states—where they come from, what they look like, how they change—don’t really matter. This is because learning algorithms are universal and the task is always the same: to teach the agent the best approach that produces the biggest reward. From a theoretical standpoint, the reward itself isn’t important, be it the speed of the robot’s movements, efficient power consumption, a click from a user, or money made at the stock exchange. The methods we used to teach robots how to run (or rather, to learn how to run themselves) could be used to distribute power generated by a nuclear power plant or advertise goods to consumers. Reinforcement learning has limitless applications.
The Yandex Prize and My Hobby
I’m delighted we received the award for a number of reasons. Firstly, personal motivation and external recognition often don’t go hand-in-hand in academia. Everyone has had their articles turned down more than once. So an award is something extraordinary, and it drives you to do better. Another important aspect is that the winners of this award receive substantial financial support, access to Yandex Toloka, and an invite to a conference. These are great motivators too.
Secondly, I didn’t think I had much of a chance of winning because I’ve had very few things published, so the award came as a pleasant surprise. The organizers must have seen me as a researcher capable of creating something interesting and useful. The number of publications wasn’t a deciding factor in their decision.
I was also delighted to tell Professor Vetrov (he wasn’t eligible for the award, although I’m sure he would have won if he had been) and my friends. Even if they don’t know much about science or research, they think it’s cool and they’re happy for me.
The award ceremony took place almost at the same time as the first concert of the ensemble I’m in, which is made up of six cellists and a pianist. We’ve got quite a wide repertoire, so there’s bound to be something for everyone to enjoy. We play everything from Shostakovich waltzes to the Game of Thrones theme. That’s another achievement I’m proud of.
The MendiT Research Lab, based at the HSE University Doctoral School of Arts and Design, will become a resident of the GES-2 Cultural Centre for a month from June 5th. The researchers will introduce members of the urban community to the lab and explore the practices of mending clothes together with them. Some of the lab participants told the HSE News Service what it's like to be searching for a method and object of research.
Doctoral Students Need the Support of Not Only Their Academic Supervisor, but Also of Other Faculty Staff
To successfully defend a doctoral dissertation, PhD candidates need not only the support of their academic supervisor and close friends and relatives, but also system-wide assistance from the university department or faculty where they study. However, HSE University researchers have found that such support can take different forms and that each has a different effect on how confident a student feels in their ability to successfully defend their dissertation. The results of their study were published in the journal Higher Education Quarterly.
Abdul-Raheem Mohammed from Ghana has recently defended a PhD dissertation in cognitive psychology at HSE University via remote procedure. Abdul-Raheem and his academic supervisor, Dmitry Lyusin, talked to the HSE News Service about their cooperation, the advantages and complications of remote PhD defence at HSE University, and the prospects of cognitive science in African countries.
International students interested in joining HSE University can apply for full-tuition scholarships from the Russian government. Applications for the 2022/23 academic year are open from March 1–11. We spoke to HSE University doctoral students about their work and about how scholarships have helped them pursue their research goals.
‘Gaining Practical Knowledge and Learning to Do a Lot in a Short Time’—The Life of an HSE PhD Student
Milica Simonovic, from Belgrade, Serbia, is a doctoral student at the HSE University Faculty of Economic Sciences. She is currently working on her doctoral thesis on corporate governance and intellectual capital disclosure. We interviewed Milica to find out more about her thesis and her experience of the university’s application process for PhD students.
Alaa N. Assaf is a doctoral student at HSE University’s Faculty of Law. In addition to doing research for his PhD thesis, he teaches undergraduates. Alaa has shared his thoughts on studying international law, life in Moscow, and what he misses about teaching in person.
HSE University continues to develop its new study format for students embarking on a research career: the Combined Master's-PhD track. This year, there will be 75 places for Master’s students on the track—two thirds more than last year. HSE Vice Rector Sergey Roshchin talks about the appeal of the combined-track option, how to enrol, and the achievements of last year’s applicants.
The British Educational Research Association, BERA, has announced the winners of its Early Career Researchers Award, founded this year to recognize emerging researchers. The new award’s first two winners are Saule Bekova, Research Fellow at theInstitute of Education of the Centre of Sociology of Higher Education, and Binwei Lu of Durham University (UK). In the announcement, the organizing committee praised them as ‘outstanding scholars’ and noted the potential significance of their research for the field of education.
At the 2021 RusCrypto Competition, HSE PhD student Anastasia Malashina (MIEM) won the student report category by the unanimous decision of the competition committee, which included the directors of the RusCrypto-2021 Association, members of the programme committee, and experts in the field of computer security. Anastasia spoke with HSE News Service about her presentation.
In 2010, HSE University launched the Full-time Advanced Doctoral Programme, an innovative programme for doctoral students at that time. At the beginning of 2021, an alumna of the programme’s first cohort became the first programme graduate to earn her Doctor of Sciences degree: Elena Vakulenko, Assistant Professor at HSE University’s Faculty of Economic Sciences. In an interview with HSE News Service, the professor discussed her academic career.