Staff Cloud Platform Engineer

Responsibilities:

• Design, build, and maintain a scalable and reliable cloud-native data lake infrastructure using Presto, Alluxio, S3, Hive and Spark technologies.
• Implement integration with Zookeeper for coordination and leader election across Datalake components.
• Configure Presto to access and query data stored in Alluxio and S3, optimizing performance and resource utilization.
• Deploy and manage Spark clusters for data processing and analytics tasks, leveraging Alluxio and S3 as storage layers.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement solutions that meet business objectives.
• Develop and maintain automated workflows for data ingestion, processing, and transformation using Spark and other relevant tools.
• Implement monitoring and alerting solutions to ensure the health and performance of the data lake infrastructure.
• Provide technical guidance and support to junior team members, fostering a culture of continuous learning and improvement.

Requirements:

• Bachelor's degree in Computer Science, Engineering, or related field; or equivalent work experience.
• 10+ years of experience as a Data Engineer or similar role, with expertise in building and managing cloud-native data lake infrastructure.
• Strong proficiency in Presto, Alluxio, S3, Spark, Hive and Zookeeper, with hands-on experience in designing and implementing solutions using these technologies.
• Experience with Spark Administration, Hive administration, Presto administration and AWS resource management.
• Experience with monitoring of distributed systems.
• Experience with infrastructure as code tools such as Terraform/Ansible for provisioning and managing cloud resources.
• Experience with container orchestration platforms such as Kubernetes for deploying and managing Datalake components.
• Solid understanding of distributed computing principles, data modelling, and database technologies.
• Experience with serverless architecture would be plus like Lambda and EMR
• Experience in Observability and Monitoring in AWS.
• Fast learner and able to troubleshoot complex scenarios while process large volumes of data (Terabytes and Petabytes)
• Excellent problem-solving skills, with the ability to troubleshoot complex issues and implement effective solutions.
• Strong communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.
• Having Databricks experience will be an added advantage.

Optional Skills:

• JVM tuning. 
• Alluxio cluster management.