Sr Test Analyst(Gen AI Platform/LLM testing)

Key Responsibilities

• Quality Strategy Ownership: Define the end-to-end testing strategy for GenAI applications(e.g., summarization, conversational flows, content generation), including selecting appropriate methodologies for agentic AI systems.
• Mentorship & Team Guidance: Lead and mentor junior QA team members, providing direction on AI-specific testing crafts, prompt engineering, and automation frameworks.
• LLM Evaluation: Validate model outputs for factual accuracy, tone, relevance, and consistency.
• Prompt Engineering & Validation: Test and refine prompt-response workflows to identify hallucinations and edge cases
• Performance Engineering: Evaluate model latency, throughput, token usage, and API response times.

Required Skills & Experience

• Experience: 5-8 years in software testing, with 13 years specialized in testing GenAI/LLM models.
• GenAI Tools: Experience with LLMs (e.g., OpenAI GPT, Claude)
• Cloud Platforms: Familiarity with AWS (Bedrock), Azure (OpenAI) etc.
• Automation Tools: Knowledge of API testing tools (Postman, REST Assured) and CI/CD tools (Jenkins, GitHub Actions), Cypress with typescript.
• QA Fundamentals: Strong understanding of Agile methodologies, functional testing, and defect tracking (Jira)
• Root Cause Analysis (RCA): Perform deep-dive analyses of production failures in AI systems to differentiate between code bugs and model behavioural issues