LLM QA

Company Description

GritWorks specializes in generating synthetic versions of real-life documents, images, and multimedia to enable enterprises to train AI models and test systems while staying compliant with data protection laws. The company was founded by AI experts with leadership experience from organizations such as Google, Infosys, and United Health Group. With a strong foundation in artificial intelligence and data privacy, GritWorks is dedicated to providing innovative solutions that empower businesses to leverage data responsibly.

Role Description

This role goes beyond traditional QA, you will define evaluation frameworks for LLM outputs, build automated validation pipelines, and work closely with AI engineers and product teams to uphold quality across the development lifecycle.

Qualifications

• 35 years of experience in software QA or test engineering
• Hands-on experience testing LLM-based or generative AI applications in production
• Experience with LLM evaluation tools (DeepEval, RAGAS, LangSmith, or equivalent)
• Strong Python programming skills for test automation and evaluation harnesses
• Solid understanding of prompt engineering, RAG pipelines, embeddings, and fine-tuning
• Experience in API testing (REST/GraphQL)
• Familiarity with CI/CD tools (Jenkins, GitHub Actions, etc.)
• Strong analytical and communication skills