Data Engineer - Databricks

Job Title: Data Engineer

Location: Coimbatore

Experience: 35 Years

Preferred Certification: Databricks Data Engineer Associate/Professional

About the Role

We are looking for a skilled Data Engineer with hands-on experience in building scalable data pipelines and working extensively on Databricks. The ideal candidate should have strong data engineering fundamentals, expertise in cloud platforms, and the ability to collaborate with cross-functional teams to deliver reliable data solutions.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Databricks and related technologies.
• Build and optimize data architectures, data models, and data workflows.
• Work with large-scale datasets to ensure efficient data ingestion, transformation, and processing.
• Implement Delta Lake best practices, including schema evolution, time travel, and ACID transactions.
• Collaborate with data scientists, analysts, and business teams to understand data requirements.
• Monitor and improve data pipeline performance and reliability.
• Ensure data security, compliance, and governance practices are followed.
• Troubleshoot and resolve issues related to data processing, orchestration, and pipeline failures.
• Automate repetitive processes to improve efficiency.

Required Skills & Qualifications

• 35 years of experience as a Data Engineer or similar role.
• Strong expertise in Databricks, Apache Spark (PySpark/Scala), and Delta Lake.
• Hands-on experience with cloud platforms (AWS, Azure, or GCP Azure preferred if using Azure Databricks).
• Proficiency in SQL and working with relational/nonrelational databases.
• Experience with data warehousing concepts and performance optimization.
• Solid understanding of ETL/ELT pipelines, data modeling, and data processing frameworks.
• Experience in Power BI for building dashboards, data visualizations, and analytics reports.
• Ability to integrate Power BI with Databricks or other data sources.
• Experience with Git, CI/CD, and data orchestration tools (Airflow, ADF, etc.).
• Strong problem-solving and analytical skills.

Preferred Qualifications

• Databricks Certification (Data Engineer Associate or Professional).
• Experience with Azure Data Factory, Azure Synapse, AWS Glue, or similar tools.
• Knowledge of MLflow, Unity Catalog, or data governance frameworks.
• Familiarity with containerization (Docker) and/or Kubernetes.

Soft Skills

• Strong communication and collaboration skills.
• Ability to work in a fast-paced and dynamic environment.
• Attention to detail with a focus on data accuracy and quality.
• Team player with a proactive attitude.