Azure Data Engineer

Senior Data Engineer

Job Summary

We are seeking a Senior Data Engineer to design and implement the data foundation for a competitive intelligence MVP. The role involves building automated ingestion pipelines, orchestrating PDF extraction workflows, and creating normalized product datasets in Bronze/Silver/Gold layers using Azure Databricks. You will work in an Azure-first environment (ADF, Databricks, Azure Functions, Azure SQL) and collaborate with AI/ML engineers and BI teams to ensure high-quality, production-ready data pipelines within timeline.

Key Responsibilities

Data Pipeline Development o Build ingestion pipelines to fetch PDFs from file stores and load into Bronze layer. o Implement orchestration using Azure Functions and integrate with Document Intelligence or Azure OpenAI for text extraction. o Develop transformation and cleaning jobs in Databricks (PySpark) for Silver layer normalization.

Data Modeling & Quality o Design schemas for Bronze/Silver/Gold layers with Delta Lake. o Implement data quality checks (completeness, deduplication, normalization). o Standardize units, sizes, and packaging for product data.

Integration & Deployment o Configure Azure Data Factory or Databricks Jobs for scheduling and monitoring. o Deploy Gold layer into Azure SQL for analytics and Power BI consumption. o Ensure secure data access and compliance with organizational standards.

Collaboration & Documentation o Work closely with AI/ML engineers for feature-ready datasets. o Provide data dictionaries, lineage documentation, and runbooks for operations.

Requirements

Must-Have Skills

8+ years in data engineering with Azure cloud experience.

Strong in Databricks (PySpark), Delta Lake, and Azure Data Factory.

Experience with Azure Functions, event-driven orchestration, and API integration.

Proficient in SQL, data modeling, and performance tuning.

Knowledge of Infrastructure as Code (IaC) for provisioning data services.

Familiarity with data quality frameworks and monitoring.

Nice-to-Have

Exposure to Azure Document Intelligence or OCR/NLP pipelines.

Experience with retail/product data normalization.

Understanding of MLOps and collaboration with AI/ML teams.

Familiarity with Power BI data shaping and optimization.