Lead Data Engineer

Job title : Lead Data Engineer

Location : Bangalore - Hybrid

Experience : 7+Years

Key Responsibilities:

• Administer and manage Databricks clusters, workspaces, and resources.
• Monitor platform health, availability, and performance.
• Develop and maintain data pipelines for ingesting, transforming, and loading data into Databricks.
• Optimize ETL processes for efficiency and scalability.
• Collaborate with data scientists and analysts to build and optimize data processing and analytics workflows using Databricks notebooks.
• Develop and implement data transformation and data analysis solutions.
• Implement and manage data lakes within Databricks for structured and unstructured data storage.
• Ensure data lake security and access controls.
• Identify and address performance bottlenecks in Databricks workloads.
• Optimize queries, data pipelines, and resource allocation for efficient data processing.
• Implement security measures to protect data within Databricks, including access controls, encryption, and auditing.
• Ensure compliance with data privacy regulations and industry standards.
• Integrate Databricks with various data sources, databases, and external systems to enable seamless data flow.
• Ensure data source connectivity and data consistency.
• Manage cluster scaling and resource allocation to meet performance and cost requirements.
• Optimise cluster configurations for specific workloads.
• Identify and resolve technical issues, errors, and anomalies in Databricks workflows.
• Provide support to users encountering problems.
• Collaborate with data engineers, data scientists, and analysts to understand data requirements and provide technical guidance.
• Conduct training sessions and knowledge sharing to empower users with Databricks capabilities.
• Create and maintain documentation for Databricks workflows, configurations, and best practices.
• Promote and enforce coding and data engineering standards.
• Monitor and optimise Databricks costs, including cluster utilisation and resource allocation.
• Ensure cost-efficient data processing.
• Stay current with Databricks updates and enhancements.
• Continuously improve Databricks workflows, processes, and infrastructure.
• Establish data governance practices within Databricks, including metadata management and data cataloguing.
• Maintain data lineage and documentation for data assets.
• Implement data quality checks and validation processes to ensure data accuracy and reliability.
• Develop and enforce data quality standards.
• Plan for scalability and resilience in Databricks infrastructure to accommodate growing data volumes and ensure high availability.
• Additional duties as assigned.

Qualifications:

• Bachelor's Degree: A bachelor's degree in computer science, information technology, data engineering, or a related field is often required. Some employers may consider equivalent work experience or relevant certifications.
• A minimum of7+years of hands-on experience working with Databricks as a data engineer, data scientist, or similar role is typically expected.
• Strong experience in data engineering, including designing and developing data pipelines, ETL processes, and data transformations.
• Familiarity with big data technologies and frameworks such as Apache Spark, Hadoop, or similar platforms commonly used in conjunction with Databricks.
• Proficiency in programming languages commonly used in Databricks, such as Python, Scala, or SQL. Knowledge of Scala is often preferred.
• Proven ability to identify and address performance bottlenecks in Databricks workloads, optimising query performance, and resource allocation.
• A commitment to staying updated with the latest Databricks updates, enhancements, and emerging technologies in the data engineering and analytics field.