Data Engineer

KPI Partners is seeking a skilled and motivated Databricks Engineer to join our dynamic team. As a Databricks Engineer, you will play a crucial role in designing, implementing, and optimizing data solutions using Databricks and Airflow. Your expertise will help drive data analytics initiatives and enhance our data-driven decision-making capabilities.

Key Responsibilities:

- Design and develop scalable data pipelines utilizing Databricks to process and analyze large datasets.

- Implement and manage workflow orchestration using Airflow.

- Collaborate with data scientists, analysts, and other stakeholders to define data requirements and ensure the availability of high-quality data.

- Optimize data workflows and ETL processes for efficiency and performance.

- Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.

- Maintain documentation related to data architecture and processes, ensuring best practices are followed.

- Stay updated with industry trends and advancements in data engineering and analytics.

Qualifications:

- Bachelor's degree in Computer Science, Engineering, or a related field.

- Proven experience working with Databricks and Airflow.

- Strong knowledge of data modeling, ETL processes, and data warehousing concepts.

- Proficiency in programming languages such as Python or Scala.

- Familiarity with cloud platforms like AWS, Azure, or Google Cloud.

- Excellent problem-solving skills and attention to detail.

- Strong communication and collaboration abilities, with a team-oriented mindset.