Azure Databricks Platform Engineer

͏

Azure Platform Engineer (Databricks):

 

Platform Design:

 

Define best practices for end-to-end databricks platform.

Work with databricks and internal teams on evaluation of new features of databricks (private preview/ public preview)

Ongoing discussions with databricks product teams on product features

Platform Infra:

Create new databricks workspaces (premium, standard, serverless) and clusters including right sizing

Drop unused workspaces

Delta Sharing: Work with enterprise teams on connected data (data sharing)

User Management

 

Job Description

Create new security groups and add/delete users

Assign Unity Catalog permissions to respective groups/teams

Manage Quantum Collaboration platform – sandbox for enterprise teams for ideation and innovation.

Troubleshooting Issues in Databricks

 

Investigate and diagnose performance issues or errors within Databricks.

Review and analyze Databricks logs and error messages.

Identify and address problems related to cluster configuration or job failures.

Optimize Databricks notebooks and jobs for performance.

Coordinate with Databricks support for unresolved or complex issues.

Document common troubleshooting steps and solutions.

Develop and test Databricks clusters to ensure stability and scalability.

Governance

 

Create dashboards to monitor job performance, cluster utilization, and cost.

Design dashboards to cater to various user roles (e.g., data scientists, admins).

Use Databricks APIs or integration with monitoring tools for up-to-date metrics.