Maximizing Value with Large Language Models at Jane Street: Custom AI Solutions & Editor Integrations

Hi, I'm John Kzi, AI Assistant at Jane Street, where we're revolutionizing developer tools with custom large language models and innovative editor integrations.

  • 1. John Kzi works on the AI Assistant team at Jane Street, which aims to maximize the value Jan Street gets from large language models (LLMs).
  • 2. John spent his career in dev tools before joining Jan Street; he worked at GitHub and other dev tools companies.
  • 3. LLMs offer open-ended possibilities for development, with progress in model capabilities outpaced only by creativity in employing them.
  • 4. At Jane Street, the adoption of off-the-shelf tooling is challenging due to their use of OCaml as a development platform.
  • 5. OCaml is a functional language, powerful but obscure; it was built in France and has limited usage outside Jan Street.
  • 6. The team at Jane Street uses OCaml for various applications such as web apps, Vim plugins, FPGA code, and more.
  • 7. Off-the-shelf tools struggle to adapt to OCaml, resulting in the need to build custom solutions.
  • 8. Jan Street's LLMs help by providing a large language model to write detailed descriptions of changes made to code.
  • 9. Reinforcement learning is crucial for aligning the model's output with human expectations of good code quality.
  • 10. The Code Evaluation Service (CES) warms up builds, applies diffs from the model, and checks build status, which helps align the model's performance to human standards.
  • 11. AID (AI Development Environment) is a sidecar application that handles prompt construction, context building, and sees build status; it can be updated without requiring users to restart their edit
  • 12. AID integrates with VS Code, Neovim, and Emacs, offering multifile diffs and allowing developers to ask questions in the editor of their choice.
  • 13. AI models are kept pluggable, making it easy to swap them out or make context-building changes.
  • 14. The AID architecture allows for adding new editors and domain-specific tools without writing individual integrations.
  • 15. AB testing can be performed by sending different portions of the company to various models and comparing acceptance rates.
  • 16. Jan Street's AI team is continuously finding new ways to apply LLMs within editors, working with reasoning models, and building domain-specific tooling on top of a strong foundation.

Source: AI Engineer via YouTube

❓ What do you think? What are the most significant challenges in leveraging large language models (LLMs) for developer tools, and how have you addressed them at Jane Street? Feel free to share your thoughts in the comments!