Technical Lead-Data Engg

Area(s) of responsibility

Job description
As a Platform Admin, will play a crucial role in designing, implementing, and maintaining our data platform using Databricks on the Microsoft Azure cloud. You’ll collaborate with cross-functional teams to ensure the scalability, reliability, and performance of our analytics and data solutions.

Experience Level:Experience Level: Level 2 (Overall 5 to 7 yrs)

Must Have Skills:
Must Have Skills: 2 to 4 years of exp in the below skill set
Snowflake, Databricks, SQL, Python, Terraform, Azure Devops

Good to Have Skills:
exp in Azure Cloud, Database services, DevOps Operation

Job Qualification / Requirements:
• Candidate working hours would be Chicago time zone (18:30 IST to 2:30 IST).
• Bachelor’s degree in Computer Science, Engineering, or related field.
• Proven experience as a Databricks administrator or similar role.
• Strong expertise in Databricks platform and its components, including workspaces, clusters, and jobs.
• Experience in configuring and optimizing Databricks clusters for big data processing.
• Familiarity with security measures and access controls within the Databricks platform.
• Understanding of data governance principles and experience with data cataloging tools like Unity Catalog.
• Experience with Infrastructure as Code (IaC) and automation tools, preferably Terraform.
• Knowledge of data catalog tools (e.g., Microsoft Purview).
• Excellent problem-solving skills and ability to work in a collaborative environment.
• Relevant certifications in Databricks or related technologies are a plus.

Job Responsibilities:
• Manage and administer the Databricks platform, including the Unity Catalog, to support data engineering and data science workflows.
• Work with Unity Catalog to organize, discover, and govern data assets within the Databricks platform.
• Maintain the data quality and consistency within the Unity Catalog.
• Collaborate with data engineers and data scientists to understand their requirements and configure the Databricks platform accordingly.
• Create and manage workspaces, clusters, and jobs in the Databricks environment.
• Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
• Monitor and manage platform costs, including optimizing resource utilization to control expenses.
• Implement security measures and access controls to ensure data privacy and protection within the Databricks platform.
• Configure and optimize Databricks clusters for efficient and scalable data processing and analytics.
• Stay up-to-date with security best practices and compliance requirements.
• Monitor the performance and health of the Databricks platform and troubleshoot any issues that arise.
• Monitor and manage platform costs, including optimizing resource utilization to control expenses.
• Setup Databricks MLOps
• Collaborate with data architects and data stewards to ensure data quality and consistency within the Unity Catalog.
• Maintain detailed documentation of configurations, procedures, and best practices.
• Design Implement and maintain Iac solutions using Terraform to provision and manage Azure Cloud resources like clusters and SQL Warehouses in Databricks.
• Build, Configure and maintain CICD pipelines using GitHub and Azure Devops for the deployment of application and infrastructure on Databricks.