Teaching a Machine to Read the Past: HSE Develops Neural Network to Decipher Manuscripts

Manuscript of playwright Aleksandr Sukhovo-Kobylin

Source: Russian State Archive of Literature and Art

Diaries and letters are an invaluable resource for humanities scholars. But what can be done when the text is impossible to read? At the HSE Faculty of Humanities, this challenge has been translated into the language of mathematics: a team of philologists, historians, and machine learning specialists has created an information system that not only recognises illegible handwriting but also helps analyse archival content.

The Background

Work with handwritten sources has long been a tradition at the faculty. A new technological stage of this began in 2019, when HSE joined the Autograph project under the leadership of Elena Penskaja (now Head of the Centre for Digital Archival Studies at the Faculty of Humanities). The project itself was launched in 2014 by a group of researchers from the Russian State Archive of Literature and Art. Almost immediately, the initiative—which enabled students, scholars, and literature enthusiasts worldwide to study digital copies of manuscripts—received support from the Pushkin House and the Russian Science Foundation.

In 2022, a group of Autograph participants decided to take the work further. They submitted a new application to the Russian Science Foundation and won a grant for the interdisciplinary, inter-university project ‘Russia’s Cultural Heritage: Intelligent Analysis and Thematic Modelling of a Corpus of Handwritten Texts.’ Historians, mathematicians, and philologists from HSE joined forces with their long-standing partners from Tomsk State University.

Elena Penskaja

Photo: HSE University

The goal was ambitious: to develop digital tools capable of transforming chaotic collections of manuscripts—diaries, letters, and ego-documents from the nineteenth and early twentieth centuries—into structured data using machine learning algorithms. The task was not merely digitisation, but the automatic identification of hidden themes, narratives, and meanings, alongside the cataloguing and intelligent analysis of archival materials.

The project formally concluded in 2025, but the research continues. Elena Penskaja and Candidate of Physical and Mathematical Sciences Nikita Lomov have created a functioning information system whose primary mission is to teach machines to read the unreadable.

How It Works: Lines, Entities, and the YOLO-HTR Neural Network

Traditional manuscript cataloguing in archives and libraries is based on organising documents into fonds, sheets, storage units, and page numbering. Digitisation adds image-based navigation, which is useful but does not solve the core problem: the text itself remains unrecognised.

The system developed at the Faculty of Humanities goes two steps further. It employs the original YOLO-HTR (You Only Look Once + Handwritten Text Recognition) neural network architecture, which simultaneously performs two tasks: locating lines of text on an image and deciphering them. As a result, each line of a manuscript becomes linked not only to a page number, but also to its textual content.

But this is only half the task. The key innovation is semantic navigation. Using large language models, the system identifies so-called entities within the text: not only traditional categories such as ‘persons,’ ‘locations,’ or ‘organisations,’ but also more complex ones such as ‘state of health,’ ‘political event,’ or ‘reflection.’ A user can click on any entity and instantly access all lines and pages where it is mentioned. This transforms an archive from a mere stack of images into an interconnected knowledge base with bidirectional cross-references.

‘We achieve content-based archive organisation,’ explained Nikita Lomov. ‘From subjects of interest, users can navigate directly to specific lines and pages, while each page and its lines provide a line-by-line list of referenced entities.’

Sukhovo-Kobylin’s Diaries: A Challenge That Lasted 40 Years

One of the most striking case studies involves the diaries of playwright Aleksandr Sukhovo-Kobylin (1817–1903). A mysterious figure, he was once suspected of murdering his French lover, wrote three plays that entered the Russian literary canon, and published almost none of his diaries during his lifetime.

Despite their impressive volume, the diaries themselves have only been partially published. Deciphering the published portion took around 40 years, and even that edition contains omissions and inaccuracies. Sukhovo-Kobylin’s handwriting is so illegible that it can easily confound an untrained reader.

Manuscript of playwright Aleksandr Sukhovo-Kobylin

Source: Russian State Archive of Literature and Art

The Faculty of Humanities team uploaded 380 pages of diaries into the system—more than 10,000 lines of text, around 5,000 of which had existing published transcriptions used to train the neural network. By comparison, the system recognises the handwriting of Fyodor Litke (Friedrich Benjamin von Lütke) and Modest Korf with an error rate of just 3–5% at the letter level. For Aleksandr Sukhovo-Kobylin, the error rate rises to 10% for letters and 28% for words.

Even so, the developers emphasise that this result represents a major breakthrough for researchers. Most errors can be corrected easily, while the text becomes highly readable in places where scholars previously had to spend several minutes deciphering each word.

Dialogue with the Machine: How to Question an Archive

Modern large language models (such as ChatGPT, DeepSeek, Gemini, and Claude) already allow users to interact with them almost as though they were speaking to a human interlocutor. The Faculty of Humanities developers have gone a step further by adapting this format specifically for archival work.

Researchers can formulate queries in natural language—for example, ‘show all mentions of illnesses in the 1850s’ or ‘identify journeys with routes and companions’—and the system will return not a continuous text, but a structured dataset with fields suitable for further analysis. This makes it possible to track the temporal dynamics of references, identify co-occurrences of entities, and reconstruct social networks, conflicts, and patterns of movement.

‘Much depends here on the wishes and ambitions of our colleagues in philology,’ said Nikita Lomov. ‘It is precisely their research interests that will determine new types of supported queries and drive the expansion of our system’s capabilities.’

Scaling Up and the Scholarly Community

The developers envision two parallel paths for the system’s future development.

The extensive path involves expanding data volumes by creating similar systems for other collections of ego-documents, particularly those for which textual transcriptions already exist.

The intensive path focuses on improving the algorithms themselves: reducing recognition errors, lowering the need for annotated data, and achieving more precise entity extraction even when transcriptions are imperfect.

However, the most important condition for success is the emergence of a community of engaged users. At present, the system operates primarily in a research mode. To seriously consider large-scale expansion, hundreds of active researchers, historians, philologists, and students are needed—not merely to observe, but to formulate queries, propose new categories of entities, and test hypotheses.

‘We would like to see a real community form around systems like this,’ said Nikita Lomov. ‘Only when the number of genuinely interested users reaches into the hundreds can the question of scaling be seriously addressed.’

The information system in Russian is already available online (currently demonstrated through the example of Sukhovo-Kobylin’s diaries).

The project continues under HSE University’s 2026 Fundamental Research Programme (‘Language, Literature, and Culture in Historical and Social Dimensions’). Those interested in testing the system or collaborating can join through the Centre for Digital Archival Studies at the HSE Faculty of Humanities.