Databricks Engineer

Company Description

ReKnew empowers enterprises to harness the full potential of their data and digital assets. Through its innovative platform of pre-built accelerators, the company enables rapid delivery of impactful use cases such as risk management, compliance, fraud detection, and complaint resolution. By modernizing data infrastructure, ReKnew ensures secure, reliable AI adoption tailored to business needs. As a growing organization, ReKnew is committed to building a team of top talent to drive this mission forward. Must be able to join immediately.

Role Overview

As Senior Databricks Engineer, you will design, build, and optimise the Lakehouse data platform that underpins AI-driven automation across the business. You will own the medallion architecture on Delta Lake, build production-grade data pipelines, enforce governance through Unity Catalog, and ensure the data layer performs at scale for ML models, real-time analytics, and intelligent agent workflows.

Key Responsibilities

Lakehouse Architecture & Data Pipelines

• Design and implement a scalable Bronze Silver Gold medallion architecture on Delta Lake
• Build and maintain production ETL/ELT pipelines using Databricks Workflows, Delta Live Tables (DLT), and Apache Spark
• Integrate structured and unstructured data from diverse enterprise sources including ERPs, CRMs, APIs, and IoT platforms
• Optimise Databricks SQL endpoints for high-performance analytics and downstream reporting

Governance, Quality & Security

• Implement Unity Catalog for end-to-end data governance, lineage tracking, and fine-grained access control
• Define and enforce data quality rules, schema evolution policies, and pipeline SLA monitoring
• Ensure compliance with local data residency, privacy, and regulatory requirements
• Manage secrets, credentials, and sensitive data handling in line with enterprise security standards

ML & AI Enablement

• Build and maintain feature stores and curated datasets to support ML model development and inference
• Manage model lifecycle using MLflow experiment tracking, model versioning, and production registry
• Collaborate with data science and AI engineering teams to ensure low-latency, high-quality data access for models and agents
• Support vector search, embeddings pipelines, and retrieval-augmented generation (RAG) workloads where applicable

Integration & Operations

• Design and manage data integration pipelines across cloud services (Azure Synapse, Data Factory, or equivalent)
• Implement real-time and near-real-time streaming pipelines for event-driven data ingestion
• Establish comprehensive monitoring, alerting, and observability for all data infrastructure
• Maintain CI/CD pipelines for data assets using Databricks Asset Bundles, Terraform, or equivalent tooling
• Document architecture decisions, pipeline specifications, and runbooks for operational handover

Required Qualifications

• 5+ years of hands-on experience with Databricks (including Delta Lake, Unity Catalog, DLT, and MLflow)
• Strong proficiency in Apache Spark (PySpark / Scala), SQL, and Python
• Demonstrated experience building medallion architectures (Bronze/Silver/Gold) at enterprise scale
• Deep knowledge of Azure cloud services: Azure Data Factory, Synapse Analytics, Azure Storage, and Key Vault
• Experience integrating ERP systems (SAP, Oracle, or similar) and third-party APIs into a Lakehouse platform
• Solid understanding of data governance, data quality frameworks, and regulatory compliance
• Experience with CI/CD for data pipelines (Databricks Asset Bundles, Terraform, or equivalent)