Abstract: Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimise for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction.
AthenaZero is a bimanual manipulator designed to maximize control authority while minimizing inertia. By utilizing quasi-direct drive actuation and transmission remotization techniques, the system achieves an effective endpoint mass comparable to that of a human. Trading off trajectory tracking stiffness as compared to conventional high-impedance manipulators, this architecture reduces reflected inertia by an order of magnitude.
Dr Anna Breger, Project Leader, University of Cambridge
In this talk we discuss the potential of non-invasive imaging and machine learning techniques for the reconstruction of degraded medieval music notation. Our examples include manuscripts and fragments that suffer from different kinds of degradations rendering parts of the notation illegible. Such degradations may happen due to chemical or physical damage, for example from iron-gall acidity or from deliberate erasure.
Pretraining LLMs at scale is reaching its limits — not in raw benchmark performance, but in the flexibility of what we can do with the resulting model. In this talk, I will argue that the path forward requires rethinking pretraining itself, including the optimizer, the architecture, and the objective. First, I will present a surprising finding: more pretraining can make models worse downstream, harder to finetune and more fragile under quantization. We trace this catastrophic overtraining to a simple culprit: sensitivity to perturbation, which grows steadily over the course of pretraining.
Dr Charles Emogor, Dept of Computer Science and Technology
Are you an early career researcher (ECR) thinking about applying for your first grant or fellowship but are not sure where to start?
If you are interested in learning more about effective grant writing and what makes a strong application then please join us for this half day workshop.
The dominant paradigm in language modeling—scaling next-token prediction with parametric knowledge storage—delivers impressive capabilities but also fundamental limitations: brittle factual memory, inefficient parameters, and myopic reasoning. Progress requires a shift toward external memory and architectures that reason globally before committing to tokens.
Positional encodings are essential for transformer-based language models to understand sequence order, yet their influence extends far beyond simple position tracking. This talk explores the landscape of positional encoding methods in LLMs and reveals surprising insights about how these architectural choices shape model behavior.
We begin with the fundamental challenge: why attention mechanisms require explicit positional information.
Hamiltonian Monte Carlo (HMC) and its variants are among the most widely used algorithms for sampling from probability distributions. Despite their popularity, quantitative convergence guarantees for unadjusted HMC remain limited, especially in divergences that provide strong relative-density control such as KL divergence and Rényi divergence. In this talk, we establish regularization properties for unadjusted HMC via one-shot couplings, which enable Wasserstein convergence guarantees to be upgraded to guarantees in KL and Rényi divergence.
This free event is open only to members of the University of Cambridge (and affiliated institutes). Please be aware that we are unable to offer consultations outside clinic hours.
If you would like to participate, please sign up as we will not be able to offer a consultation otherwise. Please sign up through the following link: https://forms.gle/oKKFG78k4CrcE6JK6. Sign-up is possible from June 4 midday (12pm) until June 8 midday or until we reach full capacity, whichever is earlier. If you successfully signed up, we will confirm your appointment by June 10 midday.
This free event is open only to members of the University of Cambridge (and affiliated institutes). Please be aware that we are unable to offer consultations outside clinic hours.
If you would like to participate, please sign up as we will not be able to offer a consultation otherwise. Please sign up through the following link: https://forms.gle/HdHM5kKYuxcdRPzr6. Sign-up is possible from June 18 midday (12pm) until June 22 midday or until we reach full capacity, whichever is earlier. If you successfully signed up, we will confirm your appointment by June 24 midday.
