Lead Data Engineer - Scala / Azure

Job Title : Lead Data Engineer - Scala/Azure

Job location : Bengaluru

Exp : 6-12 years

Notice Period : 15-30 days

Position Overview : Looking for Software Development Engineer II position in Senior Data Engineer role with very good experience in Spark with Scala. Candidate should be able to work independently on end-to-end data engineering and data science use cases, be quick learner with some lead and mentoring experience, should have overall experience of minimum 7-8 years with minimum 5 to 6 Years in Hadoop and Spark OnPrem and Azure Cloud.

The ideal candidate will have strong expertise in Scala, Spark and SQL, building scalable and efficient data pipelines on Azure.

You will be responsible for

• Design, build, and maintain scalable ETL/ELT data pipelines using Azure Data Factory, Databricks, and Spark.
• Develop and optimize spark data workflows using Scala for large-scale data processing and transformation.
• Implement performance tuning and optimization strategies for data pipelines and Spark jobs to ensure efficient data handling.
• Collaborate with data engineers to support feature engineering, model deployment, and end-to-end data engineering workflows.
• Collaborate with cross-functional teams and stakeholder teams to understand data requirements and deliver high-quality solutions
• Ensure data quality and integrity by implementing validation, error-handling, and monitoring mechanisms.
• Work with structured and unstructured data using technologies such as Delta Lake and Parquet within a Big Data ecosystem.

You will need

Primary Skills:

• Data Engineering:
• Basic concepts of Data Engineering, Ingestion from diverse sources and file formats, Hadoop, Data Warehousing, Designing & Implementing large scale Distributed Data platforms & Data Lakes
• Building distributed platforms or services
• SQL, Spark, Query Tuning & Performance Optimization
• Advanced Scala experience (e.g Functional Programming, using Case classes, Complex Data Structures & Algorithms).
• Having python and pyspark expertise with scala being primary expertise is an added advantage.
• Experience on SOLID & Dry principles and Good Software Architecture & Design implementation experience
• Languages: Scala, Python
• Good experience in Big Data Unit, System, Integration & Regression Testing
• Build & Devops experience in GitHub Actions/Jenkins, Maven/SBT ,Github, Artifactory/Jfrog, CI/CD
• Big Data Processing: Hadoop, Sqoop, Spark and Spark Streaming
• Data Streaming: Experience on Kafka and Spark Streaming
• Experience in evaluation and implementation of Data Validation & Data Quality
• Understanding of Data Lake & Medallion Architecture
• Shell Scripting & Automation using Ansible or related Configuration management tools
• Agile processes & tools like Jira & Confluence
• Code Management tools like Git
• File formats like ORC, Avro, Parquet, Json & CSV
• Big Data Orchestration: Airflow, Spark on Kubernetes, Yarn, Oozie

• Data Engineering on Azure Cloud:
• Proficiency in Azure Data Platform-Data Factory, Databricks, Azure storages, Functions etc.,
• Strong skills in Scala and SQL for data manipulation. Experience in python is an added advantage.
• Experience with ETL/ELT pipelines and data transformations on cloud
• Familiarity with Big Data Azure cloud technologies- Azure Databricks, Azure DataFactory, Delta Lake, Cloud Governance, Security.
• Architect and implement data solutions using the right tool set in Azure services such as Azure Data Factory, Azure Databricks, and Azure Blob Storage, ADLS with right industry standards.

• Data Optimization & Performance Tuning:
• Expertise in data pipeline optimization and performance tuning.
• Experience on feature engineering and model deployment.

• Analytical & Problem-Solving:
• Strong troubleshooting and problem-solving skills.
• Experience with data quality checks and validation.

Nice-to-Have Skills:

• Familiarity with data governance, security, and compliance in hybrid environments.
• Familiarity with data governance frameworks and compliance practices.
• Building Managing Large Scale on-prem Hadoop clusters or on Cloud
• Experience with wide range of range of Big Data tools, databases & frameworks e.g Kafka Connect, Apache Arrow, Apache Iceberg
• SQL & NoSQL DB: Hive, Cassandra, Dremio, Presto
• Security & Secrets Management like Ansible Vault, Snyk, Git Secrets, Hashicorp Vault.
• Infrastructure: Hands on experience on Kubernetes, Docker and related container technologies.

General Requirements for the Role:

• Should have worked on building large scale distributed systems
• Should have lead a team in tech lead/module lead role
• Should have good mentorship experience
• Should have good communication and very good documentation skills
• Should show maturity & understand the requirement and convert to high quality technical requirements
• Should Code and Design end to end data flow and deliver on time
• Be resilient & flexible to work across multiple teams and internal teams
• Should help to Implement best practices in Data Architecture and Enterprise Software Development.
• Should have extensive experience in working in Agile Data Engineering Teams Work very closely with Engineering Manager, TPM, Product Manager and Stake Holders across functions.
• Become a quick learner and be more proactive in understanding the wider Business requirements and linking to the various other concepts in the Domain.
• Help implement better solutions independently and faster with better ownership.
• Help automating manual operational tasks and focus on creating reusable assets and propel innovation.
• Work very closely with team members and able to have healthy relationship
• Be innovative and able to come up with ideas and reusable components & frameworks
• Should be ready to Support 24x7 as per the Rota