Data Engineer (Azure Databricks)

Job description:

1. Education and Experience:

• Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field.
• Proven work experience as a Data Engineer, preferably with experience in Azure Databricks.

1. Technical Skills:

• Strong proficiency in working with Azure Databricks for data processing, data transformation, and data storage.
• Experience with big data technologies such as Apache Spark, Apache Hadoop, and data lake storage solutions (e.g., Azure Data Lake Storage Gen2).
• Proficiency in programming languages commonly used in data engineering, such as Python, Scala, or SQL.
• Knowledge of data modeling, data warehousing, and ETL (Extract, Transform, Load) processes.
• Familiarity with cloud-based data storage solutions, such as Azure SQL Database, Cosmos DB, or Azure Synapse Analytics (formerly known as Azure SQL Data Warehouse).
• Experience with data integration and data pipelines using tools like Azure Data Factory or Apache NiFi.

1. Database and Querying Skills:

• Strong understanding of relational and non-relational databases.
• Proficiency in writing efficient SQL queries for data extraction and manipulation.

1. Data Governance and Security:

• Knowledge of data governance best practices and data security measures.
• Understanding of data privacy regulations and compliance (e.g., GDPR).

1. Data Visualization:

• Familiarity with data visualization tools like Power BI or Tableau.

1. Data Quality and Testing:

• Experience with data quality assurance and data testing processes.

1. Problem-Solving and Analytical Skills:

• Ability to analyze complex data sets and troubleshoot data-related issues.

1. Communication and Collaboration:

• Strong communication skills to work with cross-functional teams, including data analysts, data scientists, and business stakeholders.