Data Quality Manager

Key Responsibilities:

• Design, develop, and optimize data pipelines using PySpark to process and analyze large datasets.
• Write complex SQL queries for data extraction, transformation, and loading (ETL).
• Work with Databricks to build and maintain collaborative and scalable data solutions.
• Implement and manage CI/CD processes for data pipeline deployments to ensure seamless and efficient integration and deployment.
• Collaborate with data scientists and business analysts to understand data requirements and deliver appropriate solutions.
• Ensure data quality, integrity, and security across all data processes.
• Monitor and troubleshoot data pipelines and workflows to resolve issues promptly.
• Continuously improve data and code quality through automation and best practices.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience with PySpark, including developing and tuning data processing applications.
• Advanced proficiency in SQL and experience in writing complex queries and optimizing them for performance.
• Hands-on experience with Databricks, including notebooks, clusters, and integration with other data tools.
• Strong understanding of CI/CD pipelines and experience with tools such as Jenkins, GitLab CI/CD, or Azure DevOps.
• Familiarity with cloud platforms (e.g., AWS, Azure, Google Cloud) and related data services.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment.

Preferred Skills:

• Knowledge of data warehousing concepts and tools (e.g., Snowflake, Redshift).
• Good to have knowledge on kedro framework.