'When We Find Something That Doesn’t Work Well Enough, We Replace It in Order to Develop a More Effective Approach'
The winners of the third annual Ilya Segalovich Award were recently announced in Moscow. Established by Yandex, the award promotes the scientific endeavours of young researchers from Russia, Belarus, and Kazakhstan in the field of Computer Science. Among this year’s winners were three HSE students, including Alexander Grishin, a Doctoral student of the Big Data and Information Retrieval School of the HSE Faculty of Computer Science. Alexander spoke to us about his work, research challenges, and why he was surprised to receive the award.
HSE University and My Research Work
I started doing research work as an undergraduate in the Faculty of Biological and Medical Physics at the Moscow Institute of Physics and Technology (MIPT). In my third and fourth years there, I used machine learning to analyse DNA nucleotide sequences. It wasn’t an easy task—I had never studied machine learning before and I didn’t have enough knowledge of biology. I was keen on the subject, but I felt I lacked hard skills.
I realized that I was more interested in the technical side of things than the biology side, and I started looking for a place where I could gain those skills. That’s how I ended up in HSE University’s ‘Data Science’ Master’s Programme at the School of Data Analysis. The first year required a lot of intensive study, so I had no time for research. Then I met Professor Dmitry Vetrov, and we’ve been doing research together ever since.
My work on Professor Vetrov’s research team started with the biology side of things. I joined Insilico Medicine, a company that synthesizes pharmaceuticals. Alongside my academic advisor Daniil Polykovsky, I worked on forecasting the properties of molecules. This would enable us to generate molecules with the same properties for medicinal purposes.
Solving a difficult task requires solving easier ones first, so we started by trying to determine whether or not a molecule had any specific properties. We never published the method we developed, as two very similar articles were released around that time and the authors of those articles used a slightly better approach. That’s when we knew that our work was sound and up-to-date, just not quick enough.
After that, I began looking for another area of research. I wanted to work with Pavel Shvechikov and luckily he was able to take me on. I wanted to explore a new field, and that ended up being reinforcement learning. Pavel and I worked together for a few years. Our team gave a presentation at the NeurIPS oral workshop and wrote an article for ICML.
Our team doesn’t specialize in reinforcement learning, so we sometimes lack the level of expertise we’d like. But this is common, as not many people work on reinforcement learning at a serious academic level. We decided to put together a team dedicated to the field. As for the commercial sector, companies have always been interested in this area, but it remains underdeveloped.
The Prize-Winning Research
The things we’re doing are quite simple. Rather than focus on specific tasks, we develop general algorithms. We take an academic approach and try to develop more effective algorithms that can cover a wide range of tasks. We also study existing algorithms and the way they work in order to uncover potential problems. To put it simply, we analyze each individual element and when we find something that doesn’t work well enough, we replace it in order to develop a more effective approach.
We received the Ilya Segalovich Award for our article about overestimation bias in reinforcement learning. After performing an action, we try to figure out whether these actions will have good or bad consequences. Previously, we have relied on very primitive tools such as understatement or artificial suppression to reduce overestimation bias. But we’ve developed a more flexible and effective tool to avoid this kind of overestimation. Samsung has recognized our research too, which was a surprise to me. I thought the company would be more interested in practical results, while our work has the academic focus of enhancing the effectiveness of reinforcement learning.
Are there traditional tasks in reinforcement learning? Yes, and they’re used to test algorithms and compare them. One example is Atari games, in which the agent sees a picture, another image of the game, and performs a certain action (controlling the agent). These games vary both visually and mechanically.
We tested our algorithms on ‘locomotor tasks’—a set of tasks in a physical simulator that models different kinds of robots, from simple robots with a couple of joints to complex humanoid ones. The agent receives signals from sensors informing it of the position, angle, and speed of each limb. Every 15 milliseconds, the agent must exert a force to change the position of the body. Then the agent must exert more force to perform a new task, such as running as fast as it can without falling.
Reinforcement learning involves working with states—things the agent sees or does depending on the state it finds itself in, or rewards it receives for performing the right actions or sequences. The great thing about it is that the specifics of these states—where they come from, what they look like, how they change—don’t really matter. This is because learning algorithms are universal and the task is always the same: to teach the agent the best approach that produces the biggest reward. From a theoretical standpoint, the reward itself isn’t important, be it the speed of the robot’s movements, efficient power consumption, a click from a user, or money made at the stock exchange. The methods we used to teach robots how to run (or rather, to learn how to run themselves) could be used to distribute power generated by a nuclear power plant or advertise goods to consumers. Reinforcement learning has limitless applications.
The Yandex Prize and My Hobby
I’m delighted we received the award for a number of reasons. Firstly, personal motivation and external recognition often don’t go hand-in-hand in academia. Everyone has had their articles turned down more than once. So an award is something extraordinary, and it drives you to do better. Another important aspect is that the winners of this award receive substantial financial support, access to Yandex Toloka, and an invite to a conference. These are great motivators too.
Secondly, I didn’t think I had much of a chance of winning because I’ve had very few things published, so the award came as a pleasant surprise. The organizers must have seen me as a researcher capable of creating something interesting and useful. The number of publications wasn’t a deciding factor in their decision.
I was also delighted to tell Professor Vetrov (he wasn’t eligible for the award, although I’m sure he would have won if he had been) and my friends. Even if they don’t know much about science or research, they think it’s cool and they’re happy for me.
The award ceremony took place almost at the same time as the first concert of the ensemble I’m in, which is made up of six cellists and a pianist. We’ve got quite a wide repertoire, so there’s bound to be something for everyone to enjoy. We play everything from Shostakovich waltzes to the Game of Thrones theme. That’s another achievement I’m proud of.
‘Studying at HSE Was a Chance for Me to Get to Know Some Supportive Seniors, Knowledgeable Professors, and Wonderful Friends’
On August 4, 2023, a pre-defence of the thesis on ‘Refugee-Host Community Conflict over Assimilation, Integration, and State Legitimacy: The Case of Rohingyas in Bangladesh’ by Md. Reza Habib will be held at HSE University. The preliminary defence will take place at a joint meeting of the HSE School of Sociology and the International Laboratory for Social Integration Research. Md. Reza Habib shared his experience of studying and preparing his PhD with the HSE News Service.
Wenrui Zhang, from China, is a recent graduate of theMaster’s in Economics and Economic Policy at the HSE UniversityFaculty of Economic Sciences. Having successfully defended his master’s thesis on the impact of COVID-19 on the incomes of vulnerable groups, Wenrui has set his sights on publishing his research and enrolling in adoctoral programme at the university. The HSE News Service interviewed Wenrui about his achievements so far and his goals for the future, and also spoke to Prof.Elena Kotyrlo, his academic supervisor.
Why do international students at HSE University decide to pursue a career in academia? What fields in economics are relevant to PhD students? Richard Asiamah from Ghana and Alesya Bukreeva from Uzbekistan are current master’s students at the Faculty of Economic Sciences, but have already received scholarships from the faculty to continue their studies as doctoral students at the university. Below, they share their experiences, while Denis Melnik, Academic Director of the Doctoral School of Economics, suggests some tips for the application process.
On March 16, the results of the Open Doors International Olympiad for master’s and doctoral applicants were announced. The competition is organised by the Association of Global Universities with the support of the Ministry of Education and Science of Russia and the Federal Agency for the Commonwealth of Independent States Affairs, Compatriots Living Abroad, and International Humanitarian Cooperation (Rossotrudnichestvo). HSE University is one of the Olympiad’s organisers and the site of its distributed project office.
Wai Yan Phyo Naing first came to HSE University from Myanmar to enrol in a master’s programme at the Faculty of World Economy and World Affairs in 2015. After graduation, he decided to pursue a PhD at the Doctoral School of International and Regional Studies. On February 16, 2023, he successfully defended his dissertation on ‘Evolution of Myanmar’s Policy Towards China (1988–2020)’.
Majid Sohrabi is a 28 year-old student from Iran currently enrolled in a doctoral programme at the HSE University Faculty of Computer Science. Before starting his PhD, he graduated with honours from the university’s Master of Data Science programme. In addition to studying, he also works as an assistant at the School of Data Analysis and Artificial Intelligence and a research assistant at the Laboratory for Models and Methods of Computational Pragmatics.
Ion Frecautan chose to pursue his doctoral studies at HSE University because of its reputation as a leading research university and its extensive academic infrastructure. With the latest enrolment campaign for doctoral schools now underway, we spoke to Ion about his green finance research at the Faculty of Economic Sciences, his activities as a lecturer, and more.
On February 1, 2023, HSE University will open admissions to its doctoral schools. The Doctoral School of Economics has held a webinar for prospective students, which attracted participants from various parts of the world. Materials and feedback from the webinar, as well as information on applications, are now available on the Faculty of Economic Sciences website.
The first stage of the Open Doors: Russian Scholarship Project for prospective master’s and doctoral students has concluded. The winners of the competition get an opportunity to study at Russian universities for free. HSE University is among the most popular universities on the project.
The MendiT Research Lab, based at the HSE University Doctoral School of Arts and Design, will become a resident of the GES-2 Cultural Centre for a month from June 5th. The researchers will introduce members of the urban community to the lab and explore the practices of mending clothes together with them. Some of the lab participants told the HSE News Service what it's like to be searching for a method and object of research.