Big Data - Scala Developer, Spark ETL Frameworks with Airflow knowledge

Big Data - Scala Developer, Spark ETL Frameworks with Airflow knowledge

Apache Spark

o Spark Core

o Spark SQL

o Structured Streaming

o DataFrame / Dataset APIs

o Performance tuning and optimization

Scala

o Strong hands-on coding in Scala

o Functional programming concepts

o Developing scalable data processing applications

Databricks

o Databricks notebooks

o Job/workflow creation

o Delta Lake

o Spark optimization on Databricks

o Cluster management

o Unity Catalog basics

Secondary Skills

PySpark

o ETL development using PySpark

o Data transformation and processing

o Writing batch and streaming pipelines

Apache Airflow

o Workflow orchestration

o DAG creation and scheduling

o Monitoring and dependency management

o Pipeline automation

Database Skills

SQL

o Complex queries

o Joins, CTEs, window functions

o Query optimization

RDBMS

o PostgreSQL

o MySQL

o Oracle

o SQL Server

Big Data / Data Engineering Tools

Hive

HDFS

Kafka (good to have)

Delta Lake

Parquet

Avro

Supporting Skills

ETL / ELT pipeline development

Data warehousing concepts

Linux/Unix

Shell scripting

Git / Version control

CI/CD basics

Cloud Platforms

Azure: Databricks, Data Factory, ADLS