Sr. Databricks Engineer

JD :

Administer and maintain the Databricks workspace and infrastructure.

Set up and manage clusters, pools, and libraries to support development and production workloads.

Manage user access and workspace permissions in accordance with security policies.

Monitor system health, performance metrics, and logs to ensure platform reliability.

Collaborate with data engineers, data scientists, and DevOps teams to optimize workflows and resource usage.

Automate environment management using Infrastructure as Code (IaC) tools (e.g., Terraform).

Implement CI/CD pipelines for notebook deployments using tools like Azure DevOps, GitHub Actions, or Jenkins.

Manage integrations with external data sources, cloud storage, and services (e.g., Azure, AWS, GCP).

Ensure compliance with data governance, privacy, and security standards.

Troubleshoot issues related to performance, connectivity, cluster provisioning, and job failures.