Senior Principal Consultant - Databricks Platform Architect
Genpact
5 - 10 years
Bengaluru
Posted: 9/3/2024
Job Description
Responsibilities
• Architect and design solutions to meet functional and non-functional requirements.
• Lead the design, implementation, and optimization of our Databricks platform.
• Work closely with our data engineering team to ensure that our Databricks platform is optimized for performance, scalability, and reliability.
• Develop and maintain a comprehensive understanding of our data pipeline and data architecture.
• Collaborate with other teams to ensure that our Databricks platform is integrated with our other systems and technologies.
• Develop and maintain documentation for our Databricks platform, including architecture diagrams, deployment guides, and operational procedures.
• Provide guidance and support to our data engineering team on Databricks-related issues.
• Create and review architecture and solution design artifacts.
• Evangelize re-use through the implementation of shared assets.
• Enforce adherence to architectural standards/principles, global product-specific guidelines, usability design standards, etc.
• Proactively guide engineering methodologies, standards, and leading practices.
• Guidance of engineering staff and reviews of as-built configurations during the construction phase.
• Provide insight and direction on roles and responsibilities required for solution operations.
• Identify, communicate and mitigate Risks, Assumptions, Issues, and Decisions throughout the full lifecycle.
• Considers the art of the possible, compares various architectural options based on feasibility and impact, and proposes actionable plans.
• Demonstrate strong analytical and technical problem-solving skills.
• Ability to analyze and operate at various levels of abstraction.
• Ability to balance what is strategically right with what is practically realistic.
• Growing the Data Engineering business by helping customers identify opportunities to deliver improved business outcomes, designing and driving the implementation of those solutions.
• Supporting and developing our people, including learning & development, certification & career development plans
• Providing technical governance and oversight for platform design and implementation
• Should have technical foresight to understand new technology and advancement.
• Leading team in the definition of best practices & repeatable methodologies in Cloud Data Engineering, including Data Storage, ETL, Data Integration & Migration, Data Warehousing and Data Governance
• Should have Technical Experience in Azure, AWS & GCP Cloud Data Engineering services and solutions.
• Contributing to Sales & Pre-sales activities including proposals, pursuits, demonstrations, and proof of concept initiatives
• Evangelizing the Data Engineering service offerings to both internal and external stakeholders
• Development of Whitepapers, blogs, webinars and other though leadership material
• Development of Go-to-Market and Service Offering definitions for Data Engineering
• Working with Learning & Development teams to establish appropriate learning & certification paths for their domain.
• Expand the business within existing accounts and help clients, by building and sustaining strategic executive relationships, doubling up as their trusted business technology advisor.
• Position differentiated and custom solutions to clients, based on the market trends, specific needs of the clients and the supporting business cases.
• Build new Data capabilities, solutions, assets, accelerators, and team competencies.
• Manage multiple opportunities through the entire business cycle simultaneously, working with cross-functional teams as necessary.
Minimum Qualifications
• Excellent technical architecture skills, enabling the creation of future-proof, complex global Platform solutions on Databricks.
• Excellent interpersonal communication and organizational skills are required to operate as a leading member of global, distributed teams that deliver quality services and solutions.
• Ability to rapidly gain knowledge of the organizational structure of the firm to facilitate work with groups outside of the immediate technical team.
• Knowledge and experience in IT methodologies and life cycles that will be used.
• Familiar with solution implementation/management, service/operations management, etc.
• Leadership skills can inspire others and persuade.
• Maintains close awareness of new and emerging technologies and their potential application for service offerings and products.
• Bachelor’s Degree or equivalency (CS, CE, CIS, IS, MIS, or engineering discipline) or equivalent work experience.
• Experience in a Platform architecture role using service and hosting solutions such as private/public cloud IaaS, PaaS, and SaaS platforms.
• Experience in architecting and designing technical solutions for cloud-centric solutions based on industry standards using IaaS, PaaS, and SaaS capabilities.
• Must have strong hands-on experience on various cloud services like ADF/Lambda, ADLS/S3, Security, Monitoring, Governance & Compliance.
• Must have experience to design platform on Databricks.
• hands-on Experience to design and build Databricks based solution on any cloud platform.
• hands-on experience to design and build solution powered by DBT models and integrate with databricks.
• Must be very good designing End-to-End solution on cloud platform.
• Must have good knowledge of Data Engineering concept and related services of cloud.
• Must have good experience in Python and Spark.
• Must have good experience in setting up development best practices.
• Good to have knowledge of docker and Kubernetes.
• Experience with claims-based authentication (SAML/OAuth/OIDC), MFA, RBAC, SSO etc.
• Knowledge of cloud security controls including tenant isolation, encryption at rest, encryption in transit, key management, vulnerability assessments, application firewalls, SIEM, etc.
• Experience building and supporting mission-critical technology components with DR capabilities.
• Experience with multi-tier system and service design and development for large enterprises
• Extensive, real-world experience designing technology components for enterprise solutions and defining solution architectures and reference architectures with a focus on cloud technologies.
• Exposure to infrastructure and application security technologies and approaches
• Familiarity with requirements gathering techniques.
Preferred Qualifications
• Must have designed E2E Platform architecture on Databricks covering all the aspect of data lifecycle starting from Data Ingestion, Transformation, Serve and consumption.
• Must have excellent coding skills either Python or Scala, preferably Python.
• Must have experience in Data Engineering domain
• Must have designed and implemented at least 2-3 project end-to-end in Databricks.
• Strong expertise in Apache Spark, Delta Lake, and other Databricks components for data processing and analytics.
o Delta lake
o db API 2.0
o SQL Endpoint – Photon engine
o Unity Catalog
o Security management
o Platform governance
o Data Security
• Proficiency in AWS services including but not limited to S3, EC2, IAM, VPC, EKS, Lambda, Glue, Private Link, KMS, CloudWatch, EMR etc.
• Must have knowledge of new features available in Databricks and its implications along with various possible use-case.
• Strong expertise in designing SOX compliant platform architecture.
• Must know how to manage various Databricks workspace and its integration with other applications.
• Proficient in designing and implementing Everything as a code
o Infrastructure as a code
o Configuration as a code
o Configuration as a code
o Security configuration as a code
• Must have strong expertise in designing platform with strong observability and Monitoring standards.
• Proficient in setting best practices of various DevSecOps activities including CI/CD.
• Must have knowledge of Databricks cluster optimization and its integration with various cloud services.
• Must have strong performance optimization skills to improve efficiency and reduce cost.
• Must have strong communication skills and have worked with cross platform team.
• Must have great attitude towards learning new skills and upskilling the existing skills.
• Responsible to set best practices around Databricks CI/CD.
• Must understand composable architecture to take fullest advantage of Databricks capabilities.
• Good to have Rest API knowledge.
• Good to have understanding around cost distribution.
• Good to have if worked on migration project to build Unified data platform.
• Good to have knowledge of DBT.
• Software development full lifecycle methodologies, patterns, frameworks, libraries, and tools
• Knowledge of programming and scripting languages such as JavaScript, PowerShell, Bash, SQL, Java, Python, etc.
• Experience in distilling complex technical challenges to actionable decisions for stakeholders and guiding project teams by building consensus and mediating compromises when necessary.
• Experience coordinating the intersection of complex system dependencies and interactions
• Experience in solution delivery using common methodologies especially SAFe Agile but also Waterfall, Iterative, etc.
• Demonstrated knowledge of relevant industry trends and standards
About Company
Genpact is a global professional services firm delivering digital transformation by putting digital and data to work to create competitive advantage.
Services you might be interested in
One-Shot Campaign
Reach out to ideal employees in one shot!
The intelligent campaign for reaching out to the ideal audience to whom you can ask for help (guidance or referral).