Architect - Databricks (Cloud Data)

Job Title: Architect - Databricks (Cloud Data)

Primary skills: Databricks, PySpark, and architecture solutions.

Secondary skills: Unity Catalog, StarBurst, Azure DevOps, CICD, and orchestration layers.

Mode of work: Work from office

Experience: 13+ Years of Experience

Responsibilities:

Lead migration from Native Spark to Databricks.

Architect migration to Databricks.

Build Data Governance solutions using tools like Unity Catalog and StarBurst.

Develop orchestration workflows using Databricks and ADF.

Implement CICD pipelines for Databricks in Azure DevOps.

Process near-real-time data through Auto Loader and DLT pipelines.

Design secure, scalable infrastructure for Delta Lake and Spark SQL.

Optimize Databricks environments for cost-effectiveness.

Extract complex business logic from on-prem solutions like SSIS, Informatica, Vertica, etc., into PySpark.

Build analytical dashboards on top of the Databricks Lakehouse. Requirements - Must Have:

13+ years of overall experience, with at least 4+ years as a solution architect.

Experience with Databricks, PySpark, and modern data platforms.

Expertise in cloud-native data platform architecture and solution design.

Familiarity with Agile/Scrum methodologies.

Experience in pre-sales and in the P&C insurance domain is an advantage.

Databricks Architect