Seven-Year Zapier Veteran Shares AI Integration Success Stories

Join Olmo as he shares the story of Zapier's journey to build an end-to-end developer platform, leveraging AI and integrations to make product development faster, more accurate, and reliable.

  • 1. Malon from Zapier, not an open-source LLM.
  • 2. Olmo from Zapier, been there for over seven years.
  • 3. Apologizes for bugs introduced and welcomes bug reports.
  • 4. Anker, went through Hamill's journey, built eval tooling at his last startup.
  • 5. Led AI team at Figma, co-founded Brain Trust with Zapier as first users.
  • 6. Storytelling about what has worked for them, open to learning from the audience.
  • 7. Zapier integrates with over 7,000 apps, processing over 10 million tasks daily.
  • 8. AI-based products include Zap Builder and Zap Co-pilot.
  • 9. Brain Trust: end-to-end developer platform for world's best AI teams.
  • 10. Focused on evals, observability, and building great prompts.
  • 11. Collaborative approach between engineering and product management.
  • 12. Zapier prototype aims to reach users early, iterating based on feedback.
  • 13. AI zap Builder uses a prompt-based system for creating workflows.
  • 14. Involving product managers ensures both technical and user experience goals are met.
  • 15. Internal apps like Paths and Filters must also be supported by AI evaluations.
  • 16. Custom graders used in conjunction with logic-based tests.
  • 17. Continuous Integration (CI) for regular AI provider evaluation runs.
  • 18. Improved accuracy from 7 unit tests to over 800 with the new process.
  • 19. Next iteration: chat interface for interactive, progressive workflow creation.
  • 20. Brain Trust tracing capabilities help optimize performance in agent frameworks.
  • 21. Evaluating different models and tools for optimal co-pilot performance.
  • 22. Regression in scores after switching to GPT-4 led to further prompt engineering.
  • 23. Adopting GPT-40 significantly improved performance, but more work is needed.
  • 24. Zapier and Brain Trust collaboration: great partnership for making a quality product.

Source: AI Engineer via YouTube

❓ What do you think? What are some common challenges that organizations face when trying to implement AI-powered workflows, and how can they overcome these obstacles to achieve greater success? Feel free to share your thoughts in the comments!