Data Engineer

JOB DESCRIPTION

Position: Data Engineer

Location: Bangalore

Experience:-6 8 yrs industry experience in relevant technologies.

We are seeking a Data Engineer.

You, who have Dare to think new mindset and work as go-getter in team, are suitable to work in our learning and growth environment with several opportunities.

Key Responsibilities:

• Design, develop, and maintain the CR Analytics Hub platform, a code-first, multi-tenant data analytics solution built on Microsoft Fabric.
• Implement automated data solutions, ensuring data quality and reliability, and maintaining the platform infrastructure that enables real-time and batch analytics for multiple tenants.
• Design, implement, and maintain scalable data solutions using layered architecture patterns for streaming and batch processing.
• Develop real-time ingestion workflows using Fabric Real-Time Intelligence.
• Build multi-tenant data transformations with robust error handling, retry logic, and data quality validation throughout the workflow.
• Optimize performance of data solutions to efficiently handle high-volume IoT data across all tenant workspaces while maintaining isolation and data integrity.
• Develop and maintain automated deployment systems using Python scripts, libraries, and deployment generators that dynamically create platform artifacts from configuration data.
• Build and maintain CI/CD pipelines in Azure DevOps with multi-stage workflows, parallel tenant deployments, and quality gates.
• Manage environment-specific configurations using the golden record pattern to ensure consistent artifact deployment across all environments and enable efficient multi-tenant onboarding.
• Establish and maintain comprehensive testing frameworks including unit tests, integration tests, KQL-based data validation, and schema validation using DDL registry patterns.
• Implement parallel test execution strategies for efficient CI/CD pipelines and debug test failures in multi-tenant environments.
• Ensure all data transformations and solution changes are thoroughly validated before production deployment.
• Create and maintain technical documentation, including architecture diagrams, deployment procedures, and troubleshooting guides.
• Participate in peer code reviews, provide constructive feedback on pull requests, and collaborate on technical design decisions.
• Monitor platform performance, respond to production incidents, perform root cause analysis, and implement corrective actions.

Required Skills and Qualifications:

• Python proficiency (OOP, type hints, functional patterns) and strong SQL skills (complex queries, optimization, database concepts).
• Understanding of Git workflows (branching, PRs, conventional commits) and comprehensive testing practices (PyTest, fixtures, mocking, high coverage).
• Design and implement Azure DevOps pipelines (YAML), deployment automation, quality gates, and Python-based deployment frameworks.
• Understanding layered data architectures, streaming vs batch processing, event-driven patterns, dimensional modelling, and data warehousing concepts.
• Knowledge of KQL/query languages (trainable), Linux/Bash, Docker/DevContainers, VS Code, and documentation tools (Markdown, Mermaid).
• 3+ years of professional experience in software development or data engineering roles.
• Thorough understanding of data warehousing concepts, dimensional data modelling.
• Experience with Azure cloud services, Microsoft Fabric (ideal).
• Experience with distributed data processing frameworks (Spark, Databricks).
• Experience working in Agile teams with iterative development and collaborative delivery practices

• Bachelors degree in computer science, Software Engineering, Information Technology, or related field; or equivalent practical experience in software development or data engineering roles.

Nice to have:

• Mining industry domain knowledge.