Databricks Architect

Role Description

This is a full-time hybrid role ( Multiple roles) for a Databricks lead and Architect located in Chennai and Pondicherry, with the flexibility to work remotely on occasion. The Databricks Architect will be responsible for designing scalable architectures, developing software solutions, integrating Databricks with other platforms, and creating efficient architectural designs. Additionally, the role involves collaborating with cross-functional teams, managing projects to ensure timely delivery, and supporting the implementation of Databricks solutions that align with business goals.

Key Responsibilities:

• Be the go to person for everything related to solutioning and Architecture. Should be leading the the team.
• Lead the migration ofOracle Data Warehouse to Databricks , ensuring minimal downtime and data integrity.
• Work on data pipelines using Databricks best practices for performance, cost, and reliability, while guiding junior engineers.
• OptimizeSpark/PySparkjobs for performance and efficiency in processing large datasets.
• Implementdata quality checks, validation, and reconciliationprocesses during migration.
• Work onHadoop ecosystem (HDFS, Hive, Sqoop)for legacy data integration where needed.
• LeverageAWS cloud services (S3, EMR, EC2, Lamda, RDS, Glue)for data storage and processing.
• ImplementCI/CD pipelinesfor automated deployment of Databricks workflows.
• Ensuredata governance, security, and compliancein the cloud environment.

Mandatory Skills & Experience:

• 5+ yearsinData Engineering, with strong expertise inDatabricks, Oracle, and Data Warehousing.
• Hands-on experience inmigrating on-prem data warehouses (Oracle) to Databricks.
• Hands on expertise using key features of Databricks ( Lakeflow connect, Lakebridge, Autoloader, Unity Catalog, Spark Declarative Pipelines, Lakehouse Architecture)
• Proficiency inPySpark, SQL , and ETL/ELT frameworks.
• Strong knowledge ofHadoop ecosystem (HDFS, Hive, Sqoop).
• Experience withAWS cloud services (S3, EMR, Glue, Lambda, RDS).
• Familiarity withdata modeling, CDC, and performance tuningin Databricks.
• Good understanding oforchestration tools (Airflow, Control-M)andCI/CD pipelines.
• Proficiency in Architecture, Architectural Design, and Integration
• Strong Project Management skills to lead and ensure project milestones are met
• Ability to work with cloud-based platforms and familiarity with AI/ML and Big Data technologies
• Excellent problem-solving and analytical skills
• Strong communication and collaboration skills to work effectively in a hybrid environment
• Experience with Databricks and end-to-end deployment of data solutions is highly desirable
• Bachelors or Masters degree in Computer Science, Engineering, or a related field