Job Description:

• Overall 8-14 yrs. of Data Engineering experience with 4+ years of hands on experience in Databricks.
• Proven experience as a Databricks Architect or similar role with a deep understanding of the Databricks platform and its capabilities.
• Analyze business requirements and translate them into technical specifications for data pipelines, data lakes, and analytical processes on the Databricks platform.
• Design and architect end-to-end data solutions, including data ingestion, storage, transformation, and presentation layers, to meet business needs and performance requirements.
• Lead the setup, configuration, and optimization of Databricks clusters, workspaces, and jobs to ensure the platform operates efficiently and meets performance benchmarks.
• Manage access controls and security configurations to ensure data privacy and compliance.
• Design and implement data integration processes, ETL workflows, and data pipelines to extract, transform, and load data from various sources into the Databricks platform.
• Optimize ETL processes to achieve high data quality and reduce latency.
• Monitor and optimize query performance and overall platform performance to ensure efficient execution of analytical queries and data processing jobs.
• Identify and resolve performance bottlenecks in the Databricks environment.
• Establish and enforce best practices, standards, and guidelines for Databricks development, ensuring data quality, consistency, and maintainability.
• Implement data governance and data lineage processes to ensure data accuracy and traceability.
• Mentor and train team members on Databricks best practices, features, and capabilities.
• Conduct knowledge-sharing sessions and workshops to foster a data-driven culture within the organization.
• Will be responsible for Databricks Practice Technical/Partnership initiatives.
• Build skills in technical areas which support the deployment and integration of Databricks-based solutions to complete customer projects.