Research seminar Artificial intelligence Machine learning Data analysis
- Date: September 17, 2025
- Duration: 1:39:58
- Speakers: Sergey Nikolenko, St. Petersburg Department of the Steklov Mathematical Institute of the Russian Academy of Sciences, St. Petersburg University
Not all data is equally useful for training. Sometimes a model becomes noticeably “smarter” after seeing another 50 GB of text, and sometimes it does not improve at all. Which documents are actually the most valuable, and can we predict this in advance? These are the questions we will explore in the first seminar of the Markov Laboratory.
We will begin with simple approaches: how to build a clean training corpus, whether auxiliary models can be trained to estimate dataset quality and usefulness, and which signals are informative for this purpose (spoiler: entropy turns out to work surprisingly well).
The main focus of the seminar, however, will be influence functions (IFs). Influence functions answer the question: how would a model’s evaluation metric change if a particular document were removed from the training dataset? This seemingly simple question leads to a substantial amount of interesting mathematics—at least by deep learning standards.
While the influence of individual data points can be estimated theoretically, straightforward methods do not scale in practice. Bringing these techniques to the level of large language models requires a number of approximations and algorithmic innovations. We will discuss several foundational methods (LiSSA, Arnoldi-IF), important approximation techniques (LoGra, EK-FAC), and approaches for early-stage data valuation (LinFiK and ALinFiK).