Cloud Architect

"KPI Partners is looking for hands-on Azure Data Platform Architect with strong expertise in Databricks, DevOps (CI/CD), and IaC (Terraform/Bicep) .

This is a platform-focused architect role , not a data engineering or infrastructure admin role.

The ideal candidate must have strong hands-on expertise across Azure, Databricks, IaC, and DevOps , with the ability to design, automate, and optimize the data platform end-to-end. This is a platform-focused architectural role where the candidate acts as a Databricks Platform Owner , guiding teams, enforcing best practices, and driving automation, governance, and scalability across the environment.

The architect will own the Databricks platform setup, governance, automation, cost optimization, and best practices , while also enabling CI/CD and ADF monitoring.

Key Responsibilities

1. Databricks Platform Architecture

• Set up and configure Databricks workspaces, clusters, schemas, RBAC, and security.
• Define and enforce best practices:
• Cluster policies
• Workspace standards
• Cost control
• Job orchestration rules
• Automate Databricks (clusters, jobs, configurations) using Terraform/Bicep .
• Implement observability:
• Failed job alerts
• Cost and usage insights
• Performance reporting

2. ADF onitoring & Automation

• Review and assess pipeline quality (not develop pipelines).
• Automate extraction of ADF metrics (failures, slow pipelines, cost insights).
• Configure integration runtimes and automate ADF operations when required.

3. DevOps (CI/CD) Enablement

• Implement CI/CD for Databricks and ADF using Azure Repos/GitHub + Azure DevOps .
• Ensure controlled promotion from DEV TEST PROD.
• Not responsible for full project management pipelinesonly deployment pipelines.

4. Infrastructure-as-Code (IaC)

• Use Terraform/Bicep to automate deployments of:
• Data product resource groups
• ADF
• Key Vault
• Databricks
• Basic networking components (deep expertise not required)

5. Data Platform Governance (Long Term)

• Help define platform policies and governance strategy:
• Cataloging
• Sensitivity labels
• Data mesh / data product standards

6. Cost Optimization

• Generate Databricks usage and billing reports.
• Identify expensive or inefficient workloads.
• Recommend cost optimization strategies.

Required Skills

• Strong experience in Azure Databricks administration & platform design .
• Hands-on experience with Terraform/Bicep for Databricks/Azure automation.
• Good understanding of Azure DevOps or GitHub CI/CD .
• Familiarity with Azure services (Key Vault, ADF, Storage, Functions, etc.).
• Good scripting skills (Python/PowerShell).
• Ability to design best practices, policies, automation, and cost optimization.

Nice-to-Have

• Experience with data observability practices.
• Exposure to data governance frameworks."