Assistant Manager

Job Description

We are seeking a seasoned Data Engineer with a robust background in building scalable data pipelines and managing large-scale data architectures. With 69 years of experience, you will play a pivotal role in designing and optimizing our data ecosystem, ensuring high performance and reliability. You are an expert in the Spark ecosystem and feel at home working within Databricks to solve complex data challenges.

Key Responsibilities

• Pipeline Development: Design, develop, and maintain high-performance ETL/ELT pipelines using PySpark and Databricks.
• Experience with Rust/Vector.
• Data Modeling: Architect sophisticated data models and schema designs using SQL to support complex analytical requirements.
• Optimization: Perform performance tuning on Spark jobs and SQL queries to ensure efficient processing of massive datasets.
• Cloud Infrastructure: Leverage AWS services (such as S3, Glue, or Lambda) to integrate data sources and manage cloud-based data storage.
• Automation: Utilize Python for scripting, automation of data workflows, and building custom utility tools.
• Collaboration: Work closely with Data Scientists and Stakeholders to translate business requirements into technical data solutions.
• Mentorship: Provide technical guidance and code reviews for junior team members, ensuring best practices in data engineering.

Added Advantage

Cybersecurity Domain Expertise: Prior experience working with security datasets (e.g., SIEM logs, network traffic, threat intelligence) or building data platforms for security analytics and anomaly detection. Familiarity with data privacy standards and securing data at rest and in transit is highly preferred.