Maximizing Value with AI Assistants at Jane Street: Challenges in Utilizing Large Language Models within an OCaml-centric Development Environment

Join me as we explore the innovative approach to large language models at Jane Street, where our team is maximizing value through custom models, editor integrations, and a focus on developer tools.

1. John Kzi works at Jane Street's AI Assistant team, which maximizes the value of large language models (LLMs) for the company.
2. He has spent his career in Dev tools, previously working at GitHub and other companies.
3. Jane Street uses OCaml as their primary development platform, a functional language that is powerful but obscure and not widely used.
4. They have built their own build systems, distributed build environment, code review system (Iron), and even store their monorepo in Mercurial instead of Git.
5. The team's dream is to apply LLMs to various parts of their development flow, like resolving merge conflicts or figuring out reviewers for features.
6. John will discuss their approach to LLMs at Jane Street, focusing on custom models, editor integrations, and evaluating model performance.
7. Training models can be expensive and time-consuming but is essential for creating useful LLMs.
8. Jane Street built a Code Evaluation Service (CES) that applies the model's code changes to a base revision and checks if it compiles and passes tests.
9. CES also helps align the model's abilities with human ideas of good code during the reinforcement learning phase.
10. The real test of models is whether they work for humans, so Jane Street builds editor integrations to expose these models to developers.
11. Their AI development environment (Aid) handles prompt construction, context building, and build status; it sits as a sidecar application on developers' machines.
12. Aid's architecture allows them to swap in new models, make changes to the context-building process, add support for new editors, and integrate domain-specific tools.
13. Aid also enables A/B testing different approaches by sending portions of the company to different models and comparing acceptance rates.
14. The team is working on applying LLMs in various ways within editors and large-scale multi-agent workflows, as well as working with reasoning models more often.
15. Jane Street's approach focuses on keeping things pluggable, laying a strong foundation, and building ways for the rest of the company to add domain-specific tooling on top of it.

Source: AI Engineer via YouTube

❓ What do you think? What are your thoughts on the ideas shared in this video? Feel free to share your thoughts in the comments!