Teaching a Machine to Read the Past: HSE Develops Neural Network to Decipher Manuscripts
Diaries and letters are an invaluable resource for humanities scholars. But what can be done when the text is impossible to read? At the HSE Faculty of Humanities, this challenge has been translated into the language of mathematics: a team of philologists, historians, and machine learning specialists has created an information system that not only recognises illegible handwriting but also helps analyse archival content.
The Background
Work with handwritten sources has long been a tradition at the faculty. A new technological stage of this began in 2019, when HSE joined the Autograph project under the leadership of Elena Penskaja (now Head of the Centre for Digital Archival Studies at the Faculty of Humanities). The project itself was launched in 2014 by a group of researchers from the Russian State Archive of Literature and Art. Almost immediately, the initiative—which enabled students, scholars, and literature enthusiasts worldwide to study digital copies of manuscripts—received support from the Pushkin House and the Russian Science Foundation.
In 2022, a group of Autograph participants decided to take the work further. They submitted a new application to the Russian Science Foundation and won a grant for the interdisciplinary, inter-university project ‘Russia’s Cultural Heritage: Intelligent Analysis and Thematic Modelling of a Corpus of Handwritten Texts.’ Historians, mathematicians, and philologists from HSE joined forces with their long-standing partners from Tomsk State University.
The goal was ambitious: to develop digital tools capable of transforming chaotic collections of manuscripts—diaries, letters, and ego-documents from the nineteenth and early twentieth centuries—into structured data using machine learning algorithms. The task was not merely digitisation, but the automatic identification of hidden themes, narratives, and meanings, alongside the cataloguing and intelligent analysis of archival materials.
The project formally concluded in 2025, but the research continues. Elena Penskaja and Candidate of Physical and Mathematical Sciences Nikita Lomov have created a functioning information system whose primary mission is to teach machines to read the unreadable.
How It Works: Lines, Entities, and the YOLO-HTR Neural Network
Traditional manuscript cataloguing in archives and libraries is based on organising documents into fonds, sheets, storage units, and page numbering. Digitisation adds image-based navigation, which is useful but does not solve the core problem: the text itself remains unrecognised.
The system developed at the Faculty of Humanities goes two steps further. It employs the original YOLO-HTR (You Only Look Once + Handwritten Text Recognition) neural network architecture, which simultaneously performs two tasks: locating lines of text on an image and deciphering them. As a result, each line of a manuscript becomes linked not only to a page number, but also to its textual content.
But this is only half the task. The key innovation is semantic navigation. Using large language models, the system identifies so-called entities within the text: not only traditional categories such as ‘persons,’ ‘locations,’ or ‘organisations,’ but also more complex ones such as ‘state of health,’ ‘political event,’ or ‘reflection.’ A user can click on any entity and instantly access all lines and pages where it is mentioned. This transforms an archive from a mere stack of images into an interconnected knowledge base with bidirectional cross-references.
‘We achieve content-based archive organisation,’ explained Nikita Lomov. ‘From subjects of interest, users can navigate directly to specific lines and pages, while each page and its lines provide a line-by-line list of referenced entities.’
Sukhovo-Kobylin’s Diaries: A Challenge That Lasted 40 Years
One of the most striking case studies involves the diaries of playwright Aleksandr Sukhovo-Kobylin (1817–1903). A mysterious figure, he was once suspected of murdering his French lover, wrote three plays that entered the Russian literary canon, and published almost none of his diaries during his lifetime.
Despite their impressive volume, the diaries themselves have only been partially published. Deciphering the published portion took around 40 years, and even that edition contains omissions and inaccuracies. Sukhovo-Kobylin’s handwriting is so illegible that it can easily confound an untrained reader.
The Faculty of Humanities team uploaded 380 pages of diaries into the system—more than 10,000 lines of text, around 5,000 of which had existing published transcriptions used to train the neural network. By comparison, the system recognises the handwriting of Fyodor Litke (Friedrich Benjamin von Lütke) and Modest Korf with an error rate of just 3–5% at the letter level. For Aleksandr Sukhovo-Kobylin, the error rate rises to 10% for letters and 28% for words.
Even so, the developers emphasise that this result represents a major breakthrough for researchers. Most errors can be corrected easily, while the text becomes highly readable in places where scholars previously had to spend several minutes deciphering each word.
Dialogue with the Machine: How to Question an Archive
Modern large language models (such as ChatGPT, DeepSeek, Gemini, and Claude) already allow users to interact with them almost as though they were speaking to a human interlocutor. The Faculty of Humanities developers have gone a step further by adapting this format specifically for archival work.
Researchers can formulate queries in natural language—for example, ‘show all mentions of illnesses in the 1850s’ or ‘identify journeys with routes and companions’—and the system will return not a continuous text, but a structured dataset with fields suitable for further analysis. This makes it possible to track the temporal dynamics of references, identify co-occurrences of entities, and reconstruct social networks, conflicts, and patterns of movement.
‘Much depends here on the wishes and ambitions of our colleagues in philology,’ said Nikita Lomov. ‘It is precisely their research interests that will determine new types of supported queries and drive the expansion of our system’s capabilities.’
Scaling Up and the Scholarly Community
The developers envision two parallel paths for the system’s future development.
The extensive path involves expanding data volumes by creating similar systems for other collections of ego-documents, particularly those for which textual transcriptions already exist.
The intensive path focuses on improving the algorithms themselves: reducing recognition errors, lowering the need for annotated data, and achieving more precise entity extraction even when transcriptions are imperfect.
However, the most important condition for success is the emergence of a community of engaged users. At present, the system operates primarily in a research mode. To seriously consider large-scale expansion, hundreds of active researchers, historians, philologists, and students are needed—not merely to observe, but to formulate queries, propose new categories of entities, and test hypotheses.
‘We would like to see a real community form around systems like this,’ said Nikita Lomov. ‘Only when the number of genuinely interested users reaches into the hundreds can the question of scaling be seriously addressed.’
The information system in Russian is already available online (currently demonstrated through the example of Sukhovo-Kobylin’s diaries).
The project continues under HSE University’s 2026 Fundamental Research Programme (‘Language, Literature, and Culture in Historical and Social Dimensions’). Those interested in testing the system or collaborating can join through the Centre for Digital Archival Studies at the HSE Faculty of Humanities.
See also:
How Neural Networks Detect and Interpret Wordplay: New Insights from HSE Researchers
An international team including researchers from the HSE Faculty of Computer Science has presented KoWit-24, an annotated dataset of 2,700 Russian-language Kommersant news headlines containing wordplay. The dataset enables an assessment of how artificial intelligence detects and interprets wordplay. Experiments with five large language models show that even advanced systems still make mistakes, and that interpreting wordplay is more challenging for them than detecting it. The results were presented at the RANLP conference; the paper is available on Arxiv.org, and the dataset and the code for reproducing the experiments are available on GitHub.
Educational Programmes on Robotics and Neural Network Technologies Launch at HSE University’s Faculty of Computer Science
Every year, in response to IT industry demands, the Higher School of Economics Faculty of Computer Science launches new educational programmes while updating existing ones. In 2026, the faculty introduced Bachelor’s and Master’s degree programmes in robotics for the first time.
Group and Shuffle: Researchers at HSE University and AIRI Accelerate Neural Network Fine-Tuning
Researchers at HSE University and the AIRI Institute have proposed a method for quickly fine-tuning neural networks. Their approach involves processing data in groups and then optimally shuffling these groups to improve their interactions. The method outperforms alternatives in image generation and analysis, as well as in fine-tuning text models, all while requiring less memory and training time. The results have been presented at the NeurIPS 2024 Conference.
‘When You Have a Lot to Do, You Find Time for Everything’
Egor Churaev specialises in neural networks. In an interview for the HSE Young Scientists project, he talked about his program for determining the emotions and engagement of online conference participants, his trip to Brazil, and his sports hobbies.
Beauty in Details: HSE University and AIRI Scientists Develop a Method for High-Quality Image Editing
Researchers from theHSE AI Research Centre, AIRI, and the University of Bremen have developed a new image editing method based on deep learning—StyleFeatureEditor. This tool allows for precise reproduction of even the smallest details in an image while preserving them during the editing process. With its help, users can easily change hair colour or facial expressions without sacrificing image quality. The results of this three-party collaboration were published at the highly-cited computer vision conference CVPR 2024.
HSE University at VK Fest: VR Games and Emotion Recognition
On July 13-14, 2024, the annual large-scale VKontakte festival took place at Moscow’s Luzhniki Stadium. HSE University, as usual, participated in the event. The university's tent featured a variety of activities, including emotion recognition challenge, quizzes about artificial intelligence, IT career testing, a smile detector, VR gaming, and a blue tractor equipped with a smart sprinkler system.
Russian Researchers Improve Neural Networks' Spatial Navigation Performance
Researchers at HSE University, MISiS National University of Science and Technology, and the Artificial Intelligence Research Institute (AIRI) have developed an enhanced approach to reinforcement learning for neural networks tasked with navigation in three-dimensional environments. By using the attention mechanism, they managed to improve the performance of a graph neural network by 15%. The study results have been published in IEEE Access.
Neural Network Developed at HSE Campus in Perm Will Determine Root Cause of Stroke in Patients
Specialists at HSE Campus in Perm and clinicians at Perm City Clinical Hospital No. 4, have been collaborating to develop a neural network capable of determining the root cause of a stroke. This marks the world's first attempt to create such a system, the developers note.
HSE Researchers Teach Neural Networks to Better Detect Humour
A group of scientists from the HSE Faculty of Computer Science has conducted a study on the ability of neural networks to detect humour. It turns out that for more reliable recognition, it’s necessary to change the approach to creating datasets on which neural networks are trained. The scientists presented these results at one of the world's most important conferences on natural language processing — EMNLP 2023.
Neural Networks of Power: AI Unravels Knots and Tangles in Relationships between Humans, Elves and Hobbits
One of the most popular writers of the last century, John Ronald Reuel Tolkien, was born on January 3rd. Researchers from HSE University, AIRI and MISSIS have used machine learning to explore the social connections between the characters of his Middle-earth universe. The algorithm managed to create an accurate picture of the social structures and dynamics of the characters' relationships, providing a unique map of interactions in the epic world. The results of the work were published in IEEE Xplore.