Senior Data Engineer

About the Role

DataWeave is looking for a Senior Data Engineer to join our core engineering team. You will be responsible for building and scaling a high-performance big data stack that delivers actionable competitive intelligence to some of the world's largest retailers and brands.

We operate with a flat hierarchy, where we value technical ownership, curiosity, and the ability to solve complex data challenges in a fast-paced environment.

What Youll Do (Job Functions)

• Architecture & Design: Design and develop robust, scalable data engineering solutions for both structured and unstructured data.
• Data Pipeline Engineering: Build and maintain high-performance data pipelines and big data stacks to aggregate and process billions of data points.
• Web Scraping & Crawling: Reverse-engineer complex web structures and write scalable Python-based crawler bots to collect public data at scale.
• API Development: Build and optimize robust APIs that power our delivery points, including dashboards, visualizations, and external integrations.
• System Optimization: Troubleshoot and debug complex data completeness and quality issues while continuously scaling overall system performance.
• Technical Leadership: Collaborate across teams to align data delivery with business insights and customer reporting requirements.

What Were Looking For

• Experience: 59 years of experience in Data Engineering or Backend Development.
• Core Skills: Expert-level proficiency in Python is mandatory.
• Data Stores: Hands-on experience with MySQL, Solr, ElasticSearch, and Cassandra.
• Big Data: Proven track record of building large-scale distributed systems and RDBMS.
• Web Mining: Experience in data mining, web crawling, and handling unstructured data.
• Mindset: A "hacker" mentalityyou enjoy reverse-engineering problems and building elegant, scalable solutions.

Why DataWeave?

• Work on a platform that tracks billions of data points across the global web.
• Be part of a culture that encourages learning, experimenting, and taking total ownership of your work.
• Opportunity to work with a cutting-edge big data stack in a high-growth environment.