Azure Databricks Platform Engineer / Administrator

Job Summary

We are looking for an experienced Azure Databricks Platform Engineer / Administrator

with 510 years of experience in cloud-based big data platforms and 2+ years of hands- on Azure Databricks administration.

The ideal candidate will design, provision, and operate enterprise-scale Databricks

workspaces, enforce security, cost, and performance governance, and act as an escalation

point for data engineering and ML teams. This role focuses on delivering a reliable, secure,

and well-governed Databricks platform that accelerates product teams while meeting

enterprise compliance requirements.

Key Responsibilities :

Platform Administration & Governance

Design, provision, and operate Azure Databricks workspaces using best practices.

Create and manage clusters, cluster policies, and instance pools.

Plan and execute Databricks Runtime (DBR) upgrades.

Configure VNet-injected workspaces, network security controls, and private

endpoints.

Implement SCIM-based user and group provisioning with Azure AD.

Configure and manage secret scopes using Azure Key Vault.

Monitor jobs, audit logs, and Databricks system tables to identify platform health, performance, and cost anomalies. Unity Catalog Stewardship

Set up and manage Unity Catalog metastores, catalogs, and schemas.

Implement fine-grained access control, including object ownership and privilege delegation.

Enable row-level and column-level security.

Support data lineage, auditing, and governance using Unity Catalog features.

ETL, Streaming & Delta Live Tables Support

Partner with data engineering teams to build, operate, and optimize Spark and Delta Lake pipelines.

Support batch, streaming, and Delta Live Tables (DLT) workloads.

Assist with Databricks Workflows for orchestration and dependency management.

Troubleshoot and resolve performance and reliability issues in pipelines.

Security & Data Privacy

Integrate Azure Active Directory (AAD) single sign-on (SSO).

Enforce enterprise security standards across identity, network, and data access

layers.

Support data privacy and compliance requirements for regulated datasets.

Automation, DevOps & Infrastructure as Code

Implement Infrastructure as Code using Terraform and the Databricks Terraform

provider.

Automate workspace and Unity Catalog object management using Databricks CLI and REST APIs.

Build and maintain CI/CD pipelines for Databricks assets and configurations.

Cost Optimization & Observability

Apply resource tagging and cost-allocation strategies.

Configure auto-termination, cluster sizing, and workload optimization practices.

Provide cost visibility and recommendations to engineering and business teams.

Documentation & Knowledge Transfer

Create and maintain platform runbooks, operational SLAs, and troubleshooting

playbooks.

Deliver onboarding sessions and best-practice guidance to data engineering and ML

teams.

Act as the escalation point for complex platform-related incidents.

Required Qualifications & Skills

Experience

5+ years of experience administering cloud-based big data platforms.

2+ years of hands-on experience with Azure Databricks.

Proven experience supporting large-scale Spark ETL and ML workloads.

Technical Skills

Deep knowledge of Apache Spark 3.x and Delta Lake.

Hands-on experience with Databricks REST APIs and platform administration.

Strong knowledge of Azure services including ADLS Gen2, Azure Key Vault, VNets,

and networking concepts.

Proficiency in Python and scripting using PowerShell or Bash.

Strong experience with Terraform and Infrastructure as Code practices.

Governance & Compliance

Hands-on experience with Unity Catalog privilege model, audit tables, and lineage.

Understanding of enterprise data governance and compliance requirements.

Soft Skills

Strong incident management and troubleshooting skills.

Excellent documentation and stakeholder communication abilities.

Ability to work collaboratively with distributed engineering and ML teams.

Preferred Certifications

Databricks Platform Administrator Accreditation

Databricks Certified Data Engineer Professional

Databricks Lakehouse Fundamentals Accreditation

Microsoft Azure Administrator Associate (AZ-104)

(Preferred) Microsoft Azure Data Engineer Associate (DP203) or successor

certification

Nice to Have

Experience with Immuta, BigID, Collibra, or similar data privacy and governance

platforms.