Associate Technical Architect - Data Engineer (Azure)

Key Responsibilities:

Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks (Spark), and Azure Synapse Analytics.

Develop and optimize PySpark/Spark SQL jobs for large-scale batch and streaming data transformations.

Ingest data from various sources including Apache Kafka, REST APIs, and RDBMS, ensuring real-time or near-real-time availability.

Implement data modeling strategies (star schema, snowflake schema) for analytics consumption layers in Synapse or ADLS.

Collaborate with DevOps teams to automate deployment using CI/CD pipelines (Azure DevOps, GitHub Actions, etc.).

Monitor, troubleshoot, and optimize data workflows for performance, cost-efficiency, and reliability.

Follow coding standards, participate in peer reviews, and maintain version-controlled code in Git repositories.

Support data quality checks, logging, alerting, and observability mechanisms for production workloads.

Participate in sprint ceremonies and contribute to task estimation and delivery planning.

Must-Have Skills:

5+ years of experience in data engineering roles.

Hands-on experience with:

1. Azure Data Factory (ADF) building pipelines, triggers, linked services.
2. Azure Databricks building and managing Spark jobs in PySpark.
3. Azure Synapse Analytics data warehousing, SQL queries, workspace orchestration.
4. Apache Kafka consuming and processing real-time data streams.

Strong in SQL, Python, and Spark for data manipulation and transformation.

Exposure to CI/CD practices (Azure DevOps, Git workflows, build/release pipelines).

Understanding of data lake architecture and modern data warehousing principles.