Databricks Consultant

Key Responsibilities

• Design, implement, and optimize scalable data solutions using Snowflake and Databricks platforms to meet evolving business needs.
• Develop and maintain complex ETL/ELT pipelines integrating structured and semi-structured data from diverse sources, ensuring reliability and performance.
• Create and manage Snowflake objects including databases, schemas, tables, views, stored procedures, streams, and tasks to support robust data management and accessibility.
• Construct and optimize Databricks notebooks and Spark jobs leveraging PySpark and Spark SQL for advanced data transformation and analytics.
• Collaborate with cloud infrastructure teams to integrate data pipelines with cloud storage solutions and enterprise applications using APIs and connectors.
• Apply data modeling best practices including dimensional modeling, partitioning, and data partition strategies to enhance query performance and system scalability.
• Implement comprehensive data quality frameworks and enforce security protocols, access controls, and governance policies across the data platform.
• Engage proactively in troubleshooting, root cause analysis, and resolution of production issues to minimize system downtime and impact on business operations.
• Maintain detailed technical documentation and adhere to coding standards, testing methodologies, and release management processes to ensure consistency and maintainability.
• Drive continuous learning within the team through knowledge sharing, mentoring, and leading capability building initiatives.

Required Skills & Expertise

• Proven hands-on experience with Snowflake, including advanced querying with Snowflake SQL, managing Snowflake-specific features such as Snowpipe, Streams, Tasks, and implementing Time Travel and data sharing.
• Strong proficiency with Databricks platform, PySpark, Spark SQL, and Delta Lake architectures.
• Solid understanding of data warehousing concepts, cloud-based data architectures, and dimensional data modeling methodologies.
• Fluency in SQL and at least one programming language among Python, Scala, or Java, with practical experience in development and debugging.
• Experience in performance tuning, workload optimization, and implementing cost-effective design solutions in data engineering pipelines.
• Familiarity with DevOps practices, CI/CD pipelines, version control systems, and automated testing as they relate to data engineering workflows.
• Working knowledge of major cloud platforms such as Azure, AWS, or Google Cloud Platform, preferably with hands-on exposure to data services within these ecosystems.
• Strong analytical skills with an ability to communicate technical concepts effectively to both technical and business audiences.
• 3 to 5 years of experience in Snowflake + Databricks