Databricks - Data Engineer

Role Overview:

We are looking for a highly skilled and hands-on Senior Databricks Engineer to join our team. This is a client-facing role requiring strong technical expertise, ownership mindset, proactive communication, and the ability to build scalable enterprise-grade data solutions.

The ideal candidate should have deep experience in Databricks, PySpark, SQL/PLSQL, modern data engineering practices, CI/CD processes, and large-scale distributed data processing. The candidate should be comfortable working in fast-paced environments, troubleshooting complex data engineering problems, and driving optimization initiatives.

Key Responsibilities:

• Design, develop, and maintain scalable and high-performance data pipelines using Databricks and PySpark.
• Build and manage ingestion frameworks for batch and streaming data workloads.
• Develop enterprise-grade data solutions using Delta Lake and Medallion Architecture principles.
• Implement and manage data pipelines using DLT (Delta Live Tables) and Spark Streaming.
• Work extensively with Unity Catalog (UC) for governance and data access management.
• Optimize Spark jobs, SQL queries, and distributed processing workloads for performance and cost efficiency.
• Participate in CI/CD and release management processes using Git and Databricks Asset Bundles.
• Collaborate with business stakeholders, architects, and client teams to understand requirements and deliver robust solutions.
• Troubleshoot production issues, identify root causes, and implement sustainable fixes.
• Drive engineering best practices, code quality standards, and reusable framework development.
• Proactively communicate risks, blockers, and improvement opportunities.
• Challenge existing processes and contribute ideas to improve platform scalability, reliability, and engineering maturity.

Must-Have Skills & Experience:

Technical Skills:

• Minimum 6+ years of hands-on experience in Data Engineering (using Databricks)
• Strong hands-on experience with:
• Databricks
• PySpark
• SQL
• PLSQL
• Strong experience building large-scale and complex data pipelines.
• Hands-on experience with:
• Delta Lake
• Medallion Architecture
• Unity Catalog (UC)
• Spark Streaming
• Experience designing and developing ingestion frameworks.
• Strong troubleshooting and performance tuning skills in Spark/Databricks environments.
• Experience with CI/CD processes, Pull Request (PR) workflows, Git, and Databricks Asset Bundles.
• Experience working with distributed data processing and optimization techniques.
• Strong understanding of data engineering best practices and scalable architecture patterns.

Soft Skills:

• Excellent commitment and ownership mindset.
• Strong client-facing communication and stakeholder management skills.
• Proactive communicator with the ability to challenge status quo constructively.
• Strong analytical and problem-solving abilities.
• Ability to work independently in fast-paced delivery environments.

Nice-to-Have Skills:

• Experience implementing Data Quality frameworks and controls.
• Retail domain experience preferred.
• Data Modeling skills (dimensional modeling, warehouse modeling, etc.).
• Exposure to cloud-native data platforms (preferably Azure) and enterprise data governance practices.
• Experience working in Agile/Scrum delivery models.
• Databricks certified