Databricks Architect (Lakehouse Platform)

Company Description

ThreatXIntel is a growing Cybersecurity, IT Staffing, and Consulting company delivering end-to-end technology and security solutions.

We are hiring for our corporate client. ThreatXIntel is the official hiring partner for this requirement.

Job Overview

We are seeking a highly skilled Databricks Architect to lead the design and implementation of enterprise-scale Lakehouse data platforms.

The ideal candidate will have deep expertise in Databricks, Apache Spark, Delta Lake, and cloud-native architectures, with proven experience delivering end-to-end data engineering solutions across batch and real-time pipelines.

This role requires strong architectural leadership, hands-on development capability, and stakeholder collaboration.

Key Responsibilities

Architecture & Design

• Architect and implement enterprise-grade Lakehouse solutions using Databricks
• Design scalable cloud-based data platforms integrating multiple data sources
• Define data architecture standards, governance, and best practices

Data Engineering & Pipelines

• Build end-to-end ETL/ELT pipelines using PySpark, Scala, and SQL
• Develop batch and real-time streaming pipelines using Spark
• Design incremental loading frameworks and metadata-driven ingestion pipelines

Databricks Ecosystem

• Implement and manage:
• Delta Live Tables
• Autoloader
• Structured Streaming
• Databricks Workflows
• Integrate with orchestration tools like Apache Airflow

Data Modeling & Governance

• Design 3NF, dimensional models, and enterprise data warehouse solutions
• Implement data quality frameworks and governance standards
• Manage Unity Catalog with fine-grained security and access control

Performance Optimization

• Optimize Spark jobs, pipelines, and compute resources
• Ensure scalability, reliability, and cost efficiency

DevOps & CI/CD

• Implement CI/CD pipelines and deployment strategies
• Drive DevOps best practices for data engineering

Leadership & Collaboration

• Provide technical leadership and architectural guidance
• Collaborate with stakeholders to translate business needs into scalable solutions

Required Skills (Mandatory)

• Deep expertise in Databricks and Lakehouse architecture
• Strong experience with Apache Spark (batch & streaming)
• Hands-on experience with Delta Lake
• Strong programming skills in Python, PySpark, Scala, and SQL
• Experience with distributed data processing systems
• Strong experience in data modeling and data warehousing concepts
• Experience with real-time data processing architectures
• Hands-on experience with Unity Catalog and data governance