Apache Flume Developer (Data Engineer)

We at Coforge are hiring for a detail-oriented Data Engineer (Apache Flume) to join our team. The ideal candidate will have 4+ years of hands-on experience in development, with strong expertise in Apache Flume , data modeling , and migration . This role involves leading the migration of existing logs from SQL Server databases to an Apache Flume application and establishing a robust pipeline for daily log ingestion.

Key Responsibilities

• Log Migration:
• Design and implement solutions to extract and transfer historical logs from SQL Server to Apache Flume.
• Ensure data integrity, completeness, and consistency during migration.
• Daily Log Ingestion:
• Develop and automate pipelines to move newly generated logs from SQL Server to Flume daily.
• Monitor and maintain pipeline performance and reliability.
• System Integration & Optimization:
• Collaborate with infrastructure and application teams for seamless integration.
• Optimize data flow and troubleshoot issues in real-time.
• Documentation & Reporting:
• Maintain detailed documentation of architecture, processes, and configurations.
• Provide regular updates and reports to stakeholders.

Required Skills & Qualifications

• 3+ years of experience in data engineering or ETL development .
• Strong expertise in SQL Server and Apache Flume .
• Proficiency in scripting languages (e.g., Python , Shell ) for automation.
• Experience with log management , data pipelines , and system integration .
• Familiarity with big data ecosystems (e.g., Hadoop , Kafka ) is a plus.
• Excellent problem-solving and communication skills.

Share your resume over