Senior Data Engineer (Team Lead)

Location:In-office

Duration:Full-time (Permanent)

Experience:46 years

Mandatory Skills:

• Data pipeline architecture & development (ETL/ELT) at scale
• Advanced data modeling, data warehousing
• High proficiency in Python; building & debugging RESTful APIs
• End-to-end ownership of identifying, evaluating, and onboarding new data sources
• Data research, mining, merging & enrichment from diverse sources
• Evaluate data quality & reliability before closing partnerships
• Build & maintain web scraping/crawling infrastructure
• Design data merging, deduplication & enrichment strategies
• System architecture designing scalable, reliable data platforms
• Manage & guide a team of 24 engineers (code reviews, mentoring, setting standards)
• Architecture decision-making and technical direction for the data team
• Strong communicator cross-functional collaboration with product/business stakeholders

Mandatory Tech Stack:

• Python, Django, MongoDB
• AWS (S3, Glue, Lambda, Redshift)
• Apache Airflow, Spark/PySpark
• Elasticsearch (search-based data mining)
• Web scraping tools (Scrapy, BeautifulSoup, Selenium)
• RESTful APIs, Git/GitHub

Preferred Skills:

• Experience with data marketplace platforms (AWS Data Exchange, Snowflake Marketplace)
• Real-time/streaming data processing
• Data quality & validation frameworks
• CI/CD pipelines, infrastructure-as-code
• Agile/Scrum

Preferred Tech Stack:

• Go (Golang)
• Kafka, Docker, CloudWatch
• Third-party data APIs & marketplace integrations