Data Architect

Key Responsibilities

● Assess and balance the current in-house AI platform architecture with long-term modernization goals.

● Lead the migration of existing Apache Spark workloads to Databricks, ensuring scalability and performance.

● Design and optimize end-to-end data pipelines for both batch and real-time streaming use cases.

● Implement robust orchestration using Apache Airflow (preferably cloud-native).

● Provide architectural guidance to integrate legacy and modern data components seamlessly.

● Drive data governance, data quality, and security as core architectural pillars.

● Collaborate with data engineering, analytics, and DevOps teams to deliver scalable solutions.

Required Skills

● 12+ years in data engineering or data architecture roles, with focus on modern data platforms.

● Strong hands-on experience with Apache Spark (on-prem and Databricks environments).

● Proven experience with Databricks on Azure or AWS.

● Solid expertise in Apache Airflow for building and managing distributed data workflows.

● Strong knowledge of structured streaming frameworks (e.g., Kafka, Spark Streaming).

● Sound understanding of cloud-native data architectures (Azure, AWS, or GCP).

● Excellent problem-solving and technical leadership skills.

Preferred Qualifications

● Experience transitioning from in house data platforms to Databricks or cloud-native environments.

● Hands-on experience with Delta Lake, Unity Catalog, and performance tuning in Databricks.

● Expertise in Apache Airflow DAG design, dynamic workflows, and production troubleshooting.

● Certification in Databricks Data Engineering is highly desirable.

● Strong background in data modeling, Lakehouse architecture, and implementation patterns.

● Familiarity with telecom domain data is required.

● Experience with CI/CD pipelines, Infrastructure-as-Code (Terraform, ARM templates), and DevOps practices.

● Exposure to AI/ML model integration within real-time or batch data pipelines.