Databricks Administrator

We are looking for a Databricks Administrator with 23 years of hands-on experience to own the administration, security, and automation of our Databricks platform on AWS and Azure. You will manage workspaces, govern compute and access, maintain Unity Catalog, and drive infrastructure-as-code practices that keep our Lakehouse environment stable, secure, and cost-efficient. This is a platform administration role not a data engineering or pipeline development position.

Technical Responsibilities

Workspace Administration: Administer Databricks workspaces on AWS (E2 architecture) and Azure: user and group management, entitlements, and IP access lists. Configure and maintain SSO (SAML 2.0) and SCIM-based automated user provisioning with Entra ID or Okta. Manage workspace settings, feature flags, and runtime upgrades; track LTS versions and coordinate library dependency changes. Handle workspace onboarding and environment promotion across dev, QA, and production. Tag based RBAC implementation.

Compute & Cost Governance: Define and enforce cluster policies, instance pool configurations, and autoscaling rules across all workspaces. Monitor cluster utilization, DBU consumption, and job run costs; implement optimizations and budget alerts. Govern spot/on-demand instance usage on AWS and reserved/spot VM strategies on Azure to manage spend. Perform regular rightsizing reviews and retire unused clusters, pools, and scheduled jobs.

Unity Catalog & Data Governance: Set up and administer Unity Catalog: metastore configuration, catalog and schema structure, and fine-grained access controls. Manage data lineage visibility, audit logging, and object tagging. Configure and maintain external locations, storage credentials, and Delta Sharing for secure data access. Administer Databricks Secrets and integration with AWS Secrets Manager or Azure Key Vault.

Monitoring & Incident Management: Set up monitoring and alerting for workspace availability, job failures, cluster errors, and DBU overruns using Datadog, Grafana, or Azure Monitor. Act as first responder for platform incidents; triage, resolve, and document root cause analyses. Conduct regular health checks and produce platform availability and cost reports for stakeholders.

Infrastructure as Code (IaC): Provision and manage Databricks resources using Terraform (Databricks provider) for repeatable, version-controlled deployments. Maintain modular and reusable Terraform configurations across multiple workspaces and environments.

CI/CD & DevOps Automation: Build and maintain CI/CD pipelines (GitHub Actions or Azure DevOps) for workspace configs, cluster policies, and permission deployments. Write Python scripts for operational automation: user provisioning, cost reporting, health checks, and alerting utilities. Maintain platform runbooks, change logs, and incident documentation.

AWS & Azure Cloud Administration: Manage cloud IAM, Key Vault / Secrets Manager, storage accounts (S3, ADLS Gen2), and network configurations supporting Databricks workspaces. Ensure adherence to cloud security best practices including least-privilege access, encryption, and network isolation.

Qualifications

ZS welcomes candidates with degrees in computer science, information technology or a related discipline. A demonstrated record of success and significant experience are required. Additionally, we seek individuals with the following qualifications:

• Bachelors degree in computer science, Engineering, or a related field.
• 23 years of hands-on Databricks workspace administration on AWS and/or Azure.
• Strong understanding of Unity Catalog: metastore setup, permissions model, lineage, and external locations.
• Experience with SSO/SAML and SCIM-based user provisioning (Entra ID, Okta, or equivalent).
• Proficiency with Terraform for Databricks resource provisioning (Databricks provider).
• Working knowledge of CI/CD pipelines using GitHub Actions or Azure DevOps.
• Python scripting skills for automation, provisioning, and operational tooling.
• Hands-on experience with AWS (IAM, S3, Secrets Manager) and/or Azure (Entra ID, ADLS Gen2, Key Vault).
• Familiarity with monitoring and alerting tools: Datadog, Grafana, or Azure Monitor.
• Strong communication, documentation, and cross-functional collaboration skills.
• Self-motivated, detail-oriented, and comfortable managing multiple platform requests simultaneously.
• Databricks Certified Associate Platform Administrator certification is a plus.
• Experience administering Databricks ML workspaces, MLflow tracking servers, or Mosaic AI from a platform governance perspective is desirable.
• Familiarity with Python-based AI/ML frameworks (LangChain, OpenAI SDK) for platform integration context is advantageous.