AI Quality Engineer

JOB DESCRIPTION - AI QUALITY ENGINEER

Job Title: AI Quality Engineer

Location: Hybrid (66% On-site Panchkula)

Employment Type: Full-Time

Experience: 2 - 4 Years

Department: Engineering

About Us: Smarter.Codes is an AI Startup Studio founded 12 years ago. Our clients are startups whore building AI products. The work built by our AI Startup has got us users from notable institutions like the US Airforce, Johnson & Johnson, SGS Group, Finland Govt, and 1000s of SMBs. In addition to building startups for our clients, we are actively investing into building our product startups in the space of Developer Tools for AI Entrepreneurs, and some B2C startups.

Job Summary: We are looking for a detail-oriented and analytical Quality Engineer to help build scalable quality systems for AI-powered software products. Unlike traditional QA roles focused primarily on manual validation, this role combines automation engineering, AI evaluation, benchmarking, document fidelity testing, and product quality strategy.

The QE will be responsible for ensuring that Smarter.Codes delivers reliable, accurate, and enterprise-grade products across diverse client and internal startup environments.

Key Responsibilities:

• Design, execute, and maintain test cases for web applications, APIs, and AI-driven systems.
• Perform functional and regression testing across products and workflows.
• Validate AI-generated outputs using defined scoring systems, evaluation criteria, and benchmarking methodologies.
• Develop and maintain automation testing scripts and frameworks using Python and related tools.
• Collaborate with engineering teams to identify defects, reproduce issues, and ensure timely resolution.
• Create detailed Task Analysis Stories and structured QA documentation for workflows and edge cases.
• Analyze product behavior, identify patterns, and proactively suggest quality improvements.
• Conduct semantic and logical validation of AI responses and generated content.
• Ensure consistency, usability, reliability, and performance standards across releases.
• Contribute to continuous improvement of QA methodologies, processes, and tooling.

Required Skills & Qualifications:

Technical Skills-

• Strong understanding of Software Testing Fundamentals
• Experience with API Testing and Regression Testing strategies
• Familiarity with Automation Testing Frameworks
• Proficiency in Python scripting for testing and automation
• Experience working with and validating complex document formats
• Familiarity with CI/CD pipelines and quality validation processes
• Understanding of structured data validation techniques
• Knowledge of AI Evaluation methodologies and benchmarking concepts

Good To Have:

• Experience with document parsing libraries
• Familiarity with DOCX and PDF rendering systems
• Experience working with Enterprise SaaS products
• Exposure to AI product environments or LLM-based systems
• Knowledge of prompt engineering
• Familiarity with Selenium or Playwright

Soft Skills-

• Ownership mindset with accountability for outcomes
• Structured documentation and reporting habits
• Question Everything mindset with curiosity-driven validation
• Ability to thrive in a high-learning, fast-paced startup environment

Remuneration Range: 6 LPA to 9 LPA and above (Subject to performance in the selection process)

Why Join Us?

• Be part of building AI companies that will stand among the worlds most innovative and impactful organizations.
• Join us in shaping the future of Vertical AI, Horizontal AI, and AI Infrastructure.
• Grow from a contributor to a co-founder within our AI Startup Studio - build, own, and lead the next generation of AI companies.