Staff Software Engineer - Cloud Platform

We, the Cloud Platform Engineering team at Calix are responsible for the Platforms, Tools, and CI/CD pipelines at Calix. Our mission is to enable Calix engineers to accelerate the delivery of world-class products while ensuring the high availability, 
 

We are looking for a GCP Cloud Platform Engineer to design, implement, and manage cloud infrastructure and data pipelines using Google Cloud Platform (GCP) services like DataStream, Dataflow, Apache Flink, Apache Spark, and Dataproc. The ideal candidate will have a strong background in DevOps practices, cloud infrastructure automation, and big data technologies. You will collaborate with data engineers, developers, and operations teams to ensure seamless deployment, monitoring, and optimization of data solutions.

Responsibilities: 

-        Design and implement cloud infrastructure using IaC – Terraform etc.

-        Automate provisioning and management of Dataproc clusters, Dataflow jobs, and other GCP resources

-        Build and maintain CD pipelines for deploying data pipelines, streaming applications, and cloud infrastructure.

-        Integrate tools like GitLab CI/CD, or Cloud Build for automated testing and deployment.

-        Deploy and manage real-time and batch data pipelines using Dataflow, DataStream, and Apache Flink.

-        Ensure seamless integration of data pipelines with other GCP services like Big Query, Cloud Storage, and Kafka or Pub/Sub.

-        Implement monitoring and alerting solutions using Cloud Monitoring, Cloud Logging, and Prometheus.

-        Monitor performance, reliability, and cost of Dataproc clusters, Dataflow jobs, and streaming applications.

-        Optimize cloud infrastructure and data pipelines for performance, scalability, and cost-efficiency.

-        Implement security best practices for GCP resources, including IAM policies, encryption, and network security.

-        Ensure Observability is an integral part of the infrastructure platforms and provides adequate visibility about their health, utilization, and cost. 

-        Collaborate extensively with cross functional teams to understand their requirements; educate them through documentation/trainings and improve the adoption of the platforms/tools.  

Education and Experience:

•  7+ years of overall experience in DevOps -cloud engineering, or data engineering.
•  3+ years of experience in DevOps, cloud engineering, or data engineering.
• Proficiency in Google Cloud Platform (GCP) services, including Dataflow, DataStream, Dataproc, Big Query, and Cloud Storage.
• Strong experience with Apache Spark and Apache Flink for distributed data processing.
• Knowledge of real-time data streaming technologies (e.g., Apache Kafka, Pub/Sub).
• Familiarity with data orchestration tools like Apache Airflow or Cloud Composer.
• Expertise in Infrastructure as Code (IaC) tools like Terraform or Cloud Deployment Manager.
• Experience with CI/CD tools like Jenkins, GitLab CI/CD, or Cloud Build.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes.
• Strong scripting skills for automation (e.g., Bash, Python).
• Experience with monitoring tools like Cloud Monitoring, Prometheus, and Grafana.
• Familiarity with logging tools like Cloud Logging or ELK Stack.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.
• Ability to work in a fast-paced, agile environment.