QA Manager – AI Quality, Governance, Performance & Metrics

Role Summary

We are seeking a highly experienced QA Manager AI Quality, Governance, Performance & Metrics to lead the validation, governance, performance testing, and quality assurance of AI/ML and Generative AI systems. This role will define AI validation frameworks and standards, lead AI validation and observability engineers, and ensure AI solutions are compliant, robust, high-performing, and production-ready. The ideal candidate will bring strong QA leadership experience combined with hands-on exposure to validating AI/ML and GenAI solutions in enterprise environments.

Key Responsibilities

• Define and implement AI validation frameworks, quality standards, and testing strategies for AI/ML and GenAI solutions.
• Lead and mentor AI Validation Engineers and AI Observability Engineers.
• Execute AI governance checks to ensure compliance, bias mitigation, drift detection, robustness, and responsible AI practices.
• Own and track AI performance KPIs and quality metrics (accuracy, latency, reliability, drift, hallucination rates, etc.).
• Drive release readiness, quality gates, and sign-offs for AI-enabled products and platforms.
• Partner with Product, Engineering, MLOps, Data Science, Security, and Compliance teams to ensure high-quality AI delivery.
• Establish best practices for AI test automation, performance testing, monitoring, and reporting.
• Ensure continuous improvement of AI quality processes through data-driven insights and metrics.

Required Qualifications & Experience

• Bachelors or masters degree in computer science, Information Technology, or a related field.
• 1012 years of experience in QA, AI validation, performance engineering, or quality engineering.
• Hands-on experience validating AI/ML or Generative AI systems in production environments.
• Experience working in regulated domains (e.g., BFSI, Healthcare, Insurance, Pharma) is preferred.
• Strong experience working in Agile/DevOps environments.

Technical Skills

QA / Test Automation:

Selenium, PyTest, Robot Framework, Postman, REST API testing, JMeter

AI / ML Fundamentals:

Understanding of model training, validation, and evaluation metrics (Precision, Recall, Accuracy, F1, ROC)

Generative AI Frameworks:

OpenAI APIs, LangChain, Hugging Face Transformers, or equivalent LLM frameworks

Data Quality & Validation:

Data preparation, cleansing, and validation using Python (pandas, NumPy) or Spark

Performance & Non-functional Testing:

Load and stress testing of AI inference APIs and model endpoints

MLOps / CI-CD Tools:

Jenkins, Docker, Kubernetes, MLflow, GitHub Actions

Monitoring & Logging:

APM tools such as Dynatrace, Grafana, Prometheus; anomaly detection and drift monitoring

Cloud Platforms:

AWS, Azure ML, or Google Vertex AI; AI environment configuration and management

Scripting:

Python (preferred), SQL scripting for data validation

Model Validation Tools:

BLEU/ROUGE scoring, embedding similarity measurement, factual accuracy scoring tools

Behavioural & Leadership Skills

• Strong analytical and problem-solving skills with a data-driven mindset.
• Excellent communication and documentation skills for stakeholder reporting.
• Proven ability to mentor and upskill cross-functional QA teams in AI/ML and GenAI testing practices.
• Experience working effectively within Agile/DevOps delivery models.
• Strong commitment to ethical, responsible, and compliant AI practices.