Lead Data Engineer (Databricks)

About the Role

We are looking for a highly skilled Data Engineer with strong hands-on experience in Databricks, PySpark, GCP , and cloud data infrastructure . The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and infrastructure to support analytics, reporting, and data-driven decision-making.

Key Responsibilities

• Design, build, and optimize scalable data pipelines and ETL processes using Databricks and PySpark .
• Implement and manage Unity Catalog for data governance, lineage, and access control.
• Develop and maintain infrastructure as code using Terraform for GCP services.
• Create and manage CI/CD pipelines for data engineering workflows.
• Work with GCP Services such as Cloud Build, BigQuery, Firestore , and other related tools.
• Ensure high data quality, security, and availability across all data platforms.
• Collaborate with cross-functional teams including data scientists, analysts, and DevOps engineers.
• Troubleshoot data-related issues and optimize performance.

Required Skills

• Databricks Expert hands-on experience
• Unity Catalog Strong understanding and implementation knowledge
• PySpark Advanced coding and optimization skills
• SQL Proficient in writing complex queries and performance tuning
• Terraform Experience in automating infrastructure deployments
• CI/CD Proficient in setting up automated build and deployment pipelines
• GCP Services Hands-on experience with Cloud Build, BigQuery, Firestore , etc.

Good to Have

• Knowledge of Git and Agile methodologies