Databricks Architect

About Impetus

Impetus Technologies is a digital engineering company focused on delivering expert services and products to help enterprises achieve their transformation goals. We solve the analytics, AI, and cloud puzzle, enabling businesses to drive unmatched innovation and growth.

Founded in 1991, we are cloud and data engineering leaders providing solutions to fortune 100 enterprises, headquartered in Los Gatos, California, with development centers in NOIDA, Indore, Gurugram, Bengaluru, Pune, and Hyderabad with over 3000 global team members. We also have offices in Canada and Australia and collaborate with a number of established companies, including American Express, Bank of America, Capital One, Toyota, United Airlines, and Verizon.

Role- Databricks Architect

Job Description

• We are seeking a highly skilled Databricks Architect to lead the design, implementation, and optimization of scalable cloud-based data platforms.
• This role requires deep expertise in Lakehouse architecture, Databricks, Apache Spark, and Delta Lake, along with proven experience delivering end-to-end enterprise data engineering solutions.

Roles & Responsibilities

• Architect and implement enterprise-grade Lakehouse solutions using Databricks.Design and deliver end-to-end data engineering pipelines, including batch and real-time streaming solutions.
• Lead implementation of:Cloud-based data lakehouse platforms integrating diverse data sources.
• Real-time data processing pipelines for operational and analytical use cases.
• Develop scalable ETL/ELT pipelines using PySpark, Scala, and SQL.Implement advanced data modeling solutions including 3NF, dimensional modeling, and enterprise data warehousing strategies.
• Design and build incremental data loading frameworks and metadata-driven ingestion pipelines.
• Establish data quality frameworks and governance standards.Implement and manage Unity Catalog, including fine-grained security and access controls.
• Leverage Databricks components such as:Delta Live TablesAutoloaderStructured StreamingDatabricks WorkflowsIntegration with orchestration tools (e.g., Apache Airflow)Drive CI/CD automation, deployment strategies, and DevOps best practices.
• Optimize performance of pipelines, Spark jobs, and compute resources.
• Provide architectural guidance and technical leadership across cross-functional teams.
• Engage with stakeholders and clients to translate business requirements into scalable technical solutions.
• Deep expertise in:Databricks and cloud-native storage/compute platformsApache Spark (batch & streaming)Delta Lake & Lakehouse architectureDistributed data processing systemsStrong hands-on programming skills in Python, PySpark, Scala, and SQL.