Seven-Year Zapier Veteran Shares AI Integration Success Stories
Join Olmo as he shares the story of Zapier's journey to build an end-to-end developer platform, leveraging AI and integrations to make product development faster, more accurate, and reliable.
- 1. Malon from Zapier, not an open-source LLM.
- 2. Olmo from Zapier, been there for over seven years.
- 3. Apologizes for bugs introduced and welcomes bug reports.
- 4. Anker, went through Hamill's journey, built eval tooling at his last startup.
- 5. Led AI team at Figma, co-founded Brain Trust with Zapier as first users.
- 6. Storytelling about what has worked for them, open to learning from the audience.
- 7. Zapier integrates with over 7,000 apps, processing over 10 million tasks daily.
- 8. AI-based products include Zap Builder and Zap Co-pilot.
- 9. Brain Trust: end-to-end developer platform for world's best AI teams.
- 10. Focused on evals, observability, and building great prompts.
- 11. Collaborative approach between engineering and product management.
- 12. Zapier prototype aims to reach users early, iterating based on feedback.
- 13. AI zap Builder uses a prompt-based system for creating workflows.
- 14. Involving product managers ensures both technical and user experience goals are met.
- 15. Internal apps like Paths and Filters must also be supported by AI evaluations.
- 16. Custom graders used in conjunction with logic-based tests.
- 17. Continuous Integration (CI) for regular AI provider evaluation runs.
- 18. Improved accuracy from 7 unit tests to over 800 with the new process.
- 19. Next iteration: chat interface for interactive, progressive workflow creation.
- 20. Brain Trust tracing capabilities help optimize performance in agent frameworks.
- 21. Evaluating different models and tools for optimal co-pilot performance.
- 22. Regression in scores after switching to GPT-4 led to further prompt engineering.
- 23. Adopting GPT-40 significantly improved performance, but more work is needed.
- 24. Zapier and Brain Trust collaboration: great partnership for making a quality product.
Source: AI Engineer via YouTube
❓ What do you think? What are some common challenges that organizations face when trying to implement AI-powered workflows, and how can they overcome these obstacles to achieve greater success? Feel free to share your thoughts in the comments!