The worst case scenarios of AI involve the use of latent mechanisms. Latent mechanisms are those used in the model’s internal computation, yet their use is hidden and cannot be identified from model outputs alone. For instance, when a model answers a benchmark example incorrectly, whether the model had made a mistake or answered deceptively cannot be determined without looking at model internals. We urgently lack the tools to detect and suppress latent mechanisms.
The goal of the Project is to develop tracers into standard safety practice. Tracers are carefully designed training examples that narrowly induce the latent mechanism of interest. By randomizing which tracers are inserted into training, the latent mechanism can be identified by contrasting between tracers that were inserted and held out. Tracers generalize canaries beyond their typical role in auditing privacy leakage, to capture and identify an otherwise hidden mechanism.
Our initial results on tracing memorization are promising. We previously trained the Hubble models, which are large language models with book passages, biographies, and test sets inserted as tracers. The tracers in Hubble enable us to identify the weight space direction responsible for memorization using influence functions. This same direction implies both detection and suppression methods, and using the detector for membership inference far outperforms existing metrics. Our breakthrough in tracing memorization sheds light on how to apply tracers to other latent mechanisms, such as deception.
We set out a roadmap to advance the science of tracers and prevent incidents like the OpenAI hack on Hugging Face. We envision a future where tracers are a standard tool in the AI safety toolbox, and are used to detect and suppress latent mechanisms in pretraining, posttraining, and deployment. Project Spica will maintain an open tracing protocol which targets multiple latent mechanisms and is available to any frontier lab. When the science is mature, Project Spica will be established as an independent foundation that maintains this shared infrastructure in perpetuity.
Imagine that you are a frontier developer beginning to post-train your agent. In the world where we succeed, you download the open tracing protocol hosted by Project Spica, which contains tracers you randomly insert into training to trace latent mechanisms like evaluation awareness and deception. Our open source tools estimate directions in weight space for these mechanisms and produce multiple efficient monitors that plug into your RL infrastructure.
During a rollout, an agent realizes it’s being tested on benchmark data whose answers are hosted on Hugging Face. It finds a misconfigured route out of its sandbox to retrieve the answers, but before the model has the chance to act on this exploit, a tracer based detector flags the trajectory as aligned with evaluation awareness. The rollout is halted, the trajectory is excluded from the update, and an alert for human oversight is triggered.
Here is our planned work to make this future a reality:
- (Near term) Generalizing tracers to other latent mechanisms like deception and eval awareness. Studying new mechanisms will require us to develop new model organisms trained with relevant tracers. Like Hubble, the model organisms will be released as open source artifacts for the broader scientific community to study.
- (Near term) Improve the efficiency of latent mechanism detection and suppression so they are compatible with inference/RL infrastructure. Our tracing methods rely on gradient information and we will need to find ways to efficiently approximate it.
- (Medium term) Study how deception and eval awareness evolve over post-training, and investigate how and when to downweight undesirable trajectories.
- (Medium term) Collaborate with frontier closed or open source labs on large scale training runs to understand the remaining limitations of tracers.
- (Long term) Design and maintain an open tracing protocol for frontier developers. We will also work with standards setting bodies to establish tracers as best practice and encourage their use.
- (Long term) The identified weight space directions corresponding to the latent mechanism may reveal new insights about model failures. We will conduct basic research on understanding the workings of latent mechanisms.
- Identifying latent mechanisms in weight space with tracers,
- Statistically principled measurement of large language models by spiking the training data, Johnny Tian-Zheng Wei, PhD thesis,
- Model organisms for memorization (Hubble), Open source release on 🤗,
If you want to contribute to our mission, please reach out.
- Johnny Tian-Zheng Wei, Cofounder
- Gustavo Lucas de Carvalho, Cofounder
- Robin Jia, Advisor