AI & Data
In this age of disruption, organizations need to navigate the future with confidence, embracing decision making with clear, data-driven choices that deliver enterprise value in a dynamic business environment.
The AI & Data team leverages the power of data, analytics, robotics, science and cognitive technologies to uncover hidden relationships from vast troves of data, generate insights, and inform decision-making. The offering portfolio helps clients transform their business by architecting organizational intelligence programs and differentiated strategies to win in their chosen markets.
AI & Data will work with our clients to:
- Implement large-scale data ecosystems including data management, governance and the integration of structured and unstructured data to generate insights leveraging cloud-based platforms
- Leverage automation, cognitive and science-based techniques to manage data, predict scenarios and prescribe actions
- Drive operational efficiency by maintaining their data ecosystems, sourcing analytics expertise and providing As-a-Service offerings for continuous insights and improvements
Azure Databricks Data Engineer (3 to 6)
Job Title: Consultant
Job Summary: As a Azure Databricks Data engineer, you will lead and implement advanced data analytics and engineering solutions using Databricks on Azure. This role requires a deep understanding of big data technologies, cloud services, and data architecture strategies. You will be instrumental in transforming data into actionable insights that drive business decisions.
Key Responsibilities:
- Design and implement scalable, high-performance data solutions using Databricks on Azure platform
- Collaborate with cross-functional teams to integrate big data solutions with existing IT infrastructure.
- Develop and optimize data pipelines, architectures, and data sets.
- Perform data modeling, data validation, and ensure data accuracy and reliability.
- Implement machine learning algorithms and data processing workflows.
- Provide expertise in data storage solutions and manage large-scale data ingestion and transformation.
- Implement CI/CD based application development methodology using tools like Azure DevOps/Jenkins/TFS/power shell etc.
- Ensure compliance with data security and privacy policies.
- Mentor junior team members and lead project segments.
Qualifications:
Bachelors degree in Computer Science, Engineering, or related field.
3-6 years of experience in data engineering with a proven track record in using Databricks on Azure
Strong knowledge of Python, SQL, PySpark and Scala (optional)
Experience with cloud services such as cloud Databases, storage accounts ADLS Gen2, Azure Key vault, Cosmos DB, Azure Data factory, Azure Synapse is plus
Experience in building metadata driven ingestion and DQ framework using PySpark
Strong understanding of Lakehouse, Apache Spark, Delta Lake, and other big data technologies.
Experience working with data toolsets, including data warehouse, data marts, data lake, 3NF, and dimensional model
Experience in building pipelines using Delta live tables, autoloader, Databricks workflows for orchestration. Experience with Apache airflow will be plus.
Experience with Databricks Unity catalog is plus.
Experience in implementing fine grained access control using Databricks Unity catalog features is plus
Experience in performance optimization in Databricks/Apache spark
Demonstrated ability to work collaboratively in a team environment.
Excellent problem-solving and analytical skills.
Our purpose
Our people and culture
Professional development
Benefits to help you thrive
At Deloitte, we know that great people make a great organization. Our comprehensive rewards program helps us deliver a distinctly Deloitte experience that helps that empowers our professionals to thrive mentally, physically, and financiallyand live their purpose. To support our professionals and their loved ones, we offer a broad range of benefits. Eligibility requirements may be based on role, tenure, type of employment and/ or other criteria. Learn more about what working at Deloitte can mean for you.
Recruiting tips