Data Engineer

About the Company : We are seeking a talented Data Engineer with strong expertise in Databricks, specifically in Unity Catalog, PySpark, and SQL, to join our data team. Youll play a key role in building secure, scalable data pipelines and implementing robust data governance strategies using Unity Catalog.

Level : Manager/ Senior Associates

Years of experience : 6-11 years

Location : Bangalore

Notice : Immediate joiners/ Serving notice period

About the Role : Hiring for Azure Data Engineer/Architect (Databricks)

Responsibilities :

• Design and implement ETL/ELT pipelines using Databricks and PySpark.
• Work with Unity Catalog to manage data governance, access controls, lineage, and auditing across data assets.
• Develop high-performance SQL queries and optimize Spark jobs.
• Collaborate with data scientists, analysts, and business stakeholders to understand data needs.
• Ensure data quality and compliance across all stages of the data lifecycle.
• Implement best practices for data security and lineage within the Databricks ecosystem.
• Participate in CI/CD, version control, and testing practices for data pipelines.

Qualifications :

• Proven experience with Databricks and Unity Catalog (data permissions, lineage, audits).
• Strong hands-on skills with PySpark and Spark SQL.
• Solid experience writing and optimizing complex SQL queries.
• Familiarity with Delta Lake, data lakehouse architecture, and data partitioning.
• Experience with cloud platforms like Azure or AWS.
• Understanding of data governance, RBAC, and data security standards.

Preferred Skills :

• Databricks Certified Data Engineer Associate or Professional.
• Experience with tools like Airflow, Git, Azure Data Factory, or dbt.
• Exposure to streaming data and real-time processing.
• Knowledge of DevOps practices for data engineering.