GenAI Test Architect

Were hiring: GenAI Test Architect | Sasken

Join Sasken to shape quality engineering for Generative AI products (RAG systems, AI agents, LLM-powered applications). Youll design scalable test architecture and evaluation frameworks to ensure accuracy, reliability, safety, and Responsible AI compliance.

• What youll do: Own end-to-end QA/test architecture for GenAI systems; build automated evaluation pipelines (e.g., RAG evaluation, continuous benchmarking); define metrics for accuracy, relevance, robustness & hallucination detection; drive data quality, governance, and Responsible AI testing (bias/toxicity/safety).
• What were looking for: Strong Test/QA Architecture + automation framework experience; solid understanding of LLMs, RAG, embeddings & vector databases; hands-on with Python and AI/LLM evaluation tooling; familiarity with CI/CD integration.

Interested? Share your resume at [ajit.nair@sasken.com] with subject GenAI Test Architect. Location: [Bengaluru/Hyderabad/Chennai/Pune] | Experience: [12-15 Years]

#Hiring #GenAI #LLM #RAG #QualityEngineering #TestArchitecture #ResponsibleAI

Job Title: GenAI Test Architect

Role Overview

We are looking for a GenAI Test Architect who combines strong Test/QA Architecture expertise with the ability to design testing frameworks and quality assurance solutions for Generative AI systems.

The ideal candidate will architect and implement AI testing strategies, frameworks, and evaluation mechanisms to ensure accuracy, reliability, safety, and responsible AI compliance for GenAI-based applications such as RAG systems, AI agents, and LLM-powered solutions.

This role requires deep understanding of software testing architecture along with emerging AI evaluation techniques, including data quality validation, model performance measurement, hallucination detection, and responsible AI governance.

Key Responsibilities

GenAI Testing Architecture

• Design and implement end-to-end QA architecture for GenAI applications.
• Build testing frameworks for LLM-based systems, including RAG pipelines, agentic workflows, and AI copilots.
• Architect automated evaluation pipelines like MLFLOW, RAGAS for AI responses and model outputs.

AI Quality Evaluation

• Define and implement metrics for AI solution accuracy, coverage, relevance, and robustness.
• Design evaluation strategies for hallucination detection, prompt robustness, and response consistency.
• Implement benchmarking and continuous evaluation frameworks for GenAI systems.

Data Quality & Governance

• Design processes to validate training data, retrieval data, and prompt datasets.
• Implement data quality checks, data lineage, and dataset validation pipelines.
• Ensure retrieval quality and grounding accuracy for RAG systems.

Responsible AI & Compliance

• Define testing frameworks to ensure Responsible AI compliance, including:
• Bias detection
• Toxicity checks
• Safety validation
• Explainability
• Ensure GenAI systems adhere to ethical AI and regulatory standards.

Test Framework Development

• Architect scalable AI testing frameworks integrating:
• LLM evaluation tools
• Automated test pipelines
• CI/CD integration
• Enable continuous testing for GenAI models and pipelines.

Collaboration

• Work closely with AI architects, data scientists, product teams, and engineering teams to define quality strategies.
• Translate business requirements into measurable AI quality metrics.
• Provide technical leadership in GenAI testing best practices.

Required Skills

Testing & QA Architecture

• Strong experience in Test Architecture and QA frameworks.
• Experience designing automation frameworks and test strategies.
• Knowledge of CI/CD testing pipelines and quality engineering practices.

Generative AI & LLM Systems

• Understanding of LLMs, RAG architectures, embeddings, and vector databases.
• Experience with GenAI evaluation techniques and LLM testing methodologies.

AI Quality & Responsible AI

• Experience defining AI accuracy, precision, recall, and evaluation metrics.
• Understanding of Responsible AI frameworks, bias detection, and model governance.

Technical Skills

• Python / ML ecosystem
• AI evaluation frameworks
• LLM orchestration frameworks such as LangChain, LangGraph, or similar
• Data validation tools and model monitoring frameworks

Preferred Experience

• Experience with AI model evaluation platforms.
• Knowledge of AI observability and monitoring tools.
• Experience testing RAG systems, AI agents, and conversational AI platforms.