QA - Azure and Databricks

Job Role - QA with Azure and Databricks

Experience Required - 7 to 14 Years

Location - Indore/ Pune/ Chennai/ Bangalore

Immediate joiners are preferred.

• Key Responsibilities
• 1) Test Strategy & Planning
• Define and maintain test strategies for data pipelines, notebooks, jobs, and APIs across Databricks & Azure.
• Translate business rules and data quality SLAs into testable acceptance criteria.
• Establish risk-based test plans across functional, integration, system, and non-functional (performance, scalability, security) testing.
• 2) Data & Pipeline Testing (Databricks / Spark / Delta)
• Develop automated tests for Spark transformations (e.g., PySpark, Spark SQL) and Delta Lake features (MERGE, time travel, schema evolution).
• Implement data quality checks (completeness, accuracy, consistency, referential integrity, drift detection).
• Validate batch and streaming pipelines, job dependencies, and SLAs.
• Create reproducible test datasets and use Delta Live Tables (if applicable) to validate expectations.
• 3) Azure Cloud QA
• Test data flows across Azure Data Factory / Synapse pipelines, Databricks Jobs, Azure Storage/ADLS, Azure SQL/SQL MI, Event Hub/Kafka, and Key Vault access patterns.
• Verify IAM/ACLs/Unity Catalog permissions, secrets handling, and network boundary controls.
• Validate infrastructure configurations across environments (Dev/Test/Prod) and support IaC validation (e.g., Bicep/Terraform plan diffs).
• 4) Automation & DevOps
• Integrate tests into CI/CD (Azure DevOps/GitHub Actions): unit, integration, data-validation, and post-deployment smoke tests.
• Implement quality gates (coverage thresholds, schema checks, contract tests, static checks like Pylint/flake8).
• Build test utilities and shared libraries for data assertions, fixtures, and synthetic data generation.
• Use feature branches, PR checks, and merge policies to enforce quality.
• 5) Observability & Reliability
• Instrument tests and pipelines with logging & metrics (e.g., Azure Monitor, Log Analytics, Databricks metrics).
• Define SLIs/SLOs for data quality and pipeline reliability; participate in incident reviews and RCA.
• Establish quality dashboards (test pass rate, DQ incidents, recovery time, flaky test rate).
• 6) Collaboration & Governance
• Partner with data engineers, platform engineers, product owners, and data stewards.
• Contribute to test data management practices and a data contract approach with upstream/downstream teams.
• Document test plans, runbooks, and traceability from requirements to tests.

• Required Qualifications
• 38+ years in QA/Test Engineering with 2+ years in data platform or Spark-based environments.
• Hands-on with Databricks: Notebooks, Jobs, Delta Lake, cluster policies.
• Strong Python for tests/utilities; solid SQL for validation and profiling.
• Experience testing ETL/ELT patterns, data partitioning, late-arriving data, slowly changing dimensions, idempotency.
• Azure experience: Data Factory, ADLS Gen2, Key Vault, Azure DevOps/GitHub.
• CI/CD for data: pipelines, artifacts, environments, approvals, test stages and post-deploy validations.
• Familiar with schema validation and API testing (REST/Databricks Jobs API).
• Clear understanding of security (RBAC/ABAC, managed identities), cost & performance considerations.

• Nice-to-Have
• Unity Catalog, data lineage, and governance testing.
• Streaming (Structured Streaming, Delta Live Tables), event-driven validation.
• Performance testing of Spark jobs (cluster sizing, shuffle hotspots, skew).
• Exposure to privacy & regulatory controls (PII masking, GDPR/CCPA, SOX).

About InfoBeans:

InfoBeans is a global digital transformation and product engineering company, enabling businesses to thrive through innovation, agility, and cutting-edge technology solutions. With over 1,700 team members across the globe, we specialize in custom software development, enterprise solutions, cloud, AI/ML, UX, automation, and digital transformation services.

At InfoBeans, we live by our core purpose of Creating WOW!for our clients, team members, and the community. Our collaborative culture, growth opportunities, and people-first approach make us one of the most trusted and rewarding workplaces.