Azure Databricks Developer

Job Role - Azure Databricks Developer/ Architect

Location - Indore/ Pune/ Chennai/ Bangalore

Experience Required - 7 to 15 Years

• 1. Advanced Spark Optimization - Optimizing DataFrame transformations to avoid shuffles and wide dependencies, Adaptive Query Execution (AQE) tuning, Handling skew, Caching & checkpointing strategies, Cluster sizing & autoscaling strategies

• 2. Delta Lake Internals & Performance Tuning - Optimizing Delta performance, Understanding Delta logs & transaction protocol, Time travel & schema evolution best practices, CDC (Change Data Capture) patterns, Multi-hop architecture (Bronze/Silver/Gold)

• 3. Databricks Workflows & Orchestration - Orchestrating ETL/ELT with Jobs & Workflows, Multi-task job pipelines (with task dependencies), Job clusters vs all-purpose clusters, Error handling & retries, CI/CD deployment using repos

• 4. Unity Catalog & Enterprise Data Governance - Fine-grained access control (schemas, tables, columns, views), Data lineage tracking, Secure data sharing across workspaces, Managing tokens, service principals & permissions models

• 5. Databricks SQL & Lakehouse Warehouses - Writing high-performance queries on the Lakehouse, Materialized views, SQL dashboards, Understanding Photon execution engine for high-speed queries

• 7. Streaming & RealTime Pipelines - Structured Streaming internals, Auto Loader for incremental ingest, Trigger types & watermarking, Handling late-arriving data, Stateful stream processing

• 8. Advanced Cluster & Compute Management - Spot vs on-demand clusters, Photon runtime vs standard, Cluster policies for cost control, SQL warehouses vs all-purpose compute, Monitoring with Ganglia / metrics dashboards
• 9. Best Practices in Lakehouse Architecture - Medallion Architecture patterns, Modular, reusable ETL code patterns, Cost optimization strategies, Data quality frameworks (e.g., expectations, constraints)

• 10. DevOps & CI/CD Integration - Git integration via Databricks Repos, Promoting code across dev/test/prod, Using the Databricks CLI, Automated deployments using Azure DevOps/GitHub Actions

• Nice to have:

• 11. Azure Data Factory - Understanding of Pipelines, Activities, and Datasets, Linked Services configuration, Integration Runtime types (AutoIR, SelfHosted IR, Azure IR), Source and Sink concepts in ADF
• ETL/ELT Data Integration - Building data ingestion pipelines (batch + incremental loads), Designing endtoend data transformation workflows, Dataflow mapping and wrangling, Implementing control flows (If, Switch, ForEach, Until)