Senior Data Engineer

JD & Role Overview:

As a Senior Data Engineer, you will be the backbone of our Martech data ecosystem. You will design, build, and optimize the data pipelines that power real-time promotions and personalized marketing journeys. We need someone who can take a business requirement from a whiteboard and turn it into a production-ready, end-to-end deliverable within our Databricks and AWS S3 environment.

Key Responsibilities:

Data Lakehouse Architecture: Design and implement robust Data Lake patterns on AWS S3 using Databricks/Delta Lake to ensure data is scalable, searchable, and secure.

ETL/ELT Development: Build end-to-end data pipelines to ingest, transform, and load data from various Martech sources, ensuring high quality and low latency.

Streaming Integration: Collaborate on the integration of Apache Flink event streams into the data lake to support real-time promotional analytics and historical auditing.

Marketing Automation Support: Facilitate data flows between the Data Lake and Braze, ensuring our marketing automation tool has the "golden record" of user data for personalized campaigns.

Performance Tuning: Optimize Spark jobs and SQL queries to manage large-scale datasets efficiently, keeping an eye on both performance and AWS cost management.

Autonomy & Delivery: Own the full development lifecyclefrom requirement gathering and data modeling to deployment and monitoringwithout needing constant hand-holding.

Technical Requirements:

Data Platforms: 5+ years of hands-on experience with Databricks (Spark, Delta Lake, Unity Catalog).

Cloud Infrastructure: Deep expertise in AWS S3, including bucket policies, storage classes, and data organization best practices.

Streaming Knowledge: Familiarity with Apache Flink or similar stream processing frameworks (e.g., Spark Streaming, Kafka Streams) for event-driven architectures.

Languages: Expert-level proficiency in Python (PySpark) and SQL.

Martech Experience: Proven experience working with marketing data or tools like Braze, Salesforce Marketing Cloud, or similar CDPs/automation platforms.

Data Modeling: Strong understanding of Medallion Architecture (Bronze/Silver/Gold) and Schema-on-read principles.

Soft Skills:

Independent Problem Solver: You dont wait for a ticket to tell you whats broken; you find the bottleneck and fix it.

Clear Communicator: You can explain complex data flows to non-technical stakeholders in the marketing team.

Delivery-Focused: You have a "get it done" attitude with a high bar for code quality and documentation.

Preferred Qualifications:

Experience with Infrastructure as Code (Terraform/CloudFormation).

Knowledge of CI/CD pipelines for data engineering (GitHub Actions, Jenkins, or Azure DevOps).

AWS Certified Data Engineer or Databricks Certified Professional.