Big Data Engineer

JD Big Data Engineer

Experience: 5+ years

Work Mode: Remote

Work Timings: 11:00 PM 7:00 AM EST

Job Summary:

We are seeking a mid-level Big Data Engineer / Support Engineer to provide operational support for Market Regulations big data processing and hybrid infrastructure. The role focuses on monitoring and maintaining high-volume batch and data pipelines in AWS, troubleshooting database and scheduling issues, automating monitoring and remediation, and maintaining enterprise Linux and Windows systems across on-premises and cloud environments. The ideal candidate combines systems administration, DevOps practices, containerization, cloud operations (AWS/Azure), and scripting to ensure reliability, availability, and serviceability for critical regulatory systems. This position requires working the night shift (11:00 PM 7:00 AM EST) with occasional weekend support.

Required Skills & Experience:

• A minimum of 5 years of related experience as a big data engineer.
• Production experience as a systems administrator or support engineer across Linux and Windows environments.
• Hands-on experience with AWS services (EC2, ECS, Lambda, CloudWatch, IAM) and exposure to Azure (VMs, Container Instances, Monitor, AD).
• Strong SQL skills with experience in Postgres, Oracle, or equivalent relational databases; ability to build and run complex queries.
• Proficient scripting skills in Python and Bash; PowerShell experience desirable.
• Experience with workload automation/schedulers such as JAMS, Broadcom AutoSys, or equivalent.
• Familiarity with containerization (Docker, ECS or Kubernetes) and CI/CD tools (Jenkins, GitLab); strong understanding of git branching and deployment pipelines.
• Demonstrated troubleshooting and debugging skills for OS, applications, and infrastructure, including patching, deployment, and server hardware maintenance.
• Experience with monitoring, observability, and performance tuning across servers, containers, and cloud resources.
• Knowledge of change management, incident and problem management, metrics reporting, and SOX/DR/BCP support.
• Strong written and verbal communication and stakeholder collaboration skills

Good to Have:

• Experience with AI-enhanced observability and automation tools.
• Experience with infrastructure-as-code and enterprise automated deployment pipelines.
• Certifications: AWS SysOps, AWS DevOps, or relevant Linux/DB certifications.
• Prior experience supporting financial services, market regulation, or other regulated environments.
• Familiarity with monitoring platforms (CloudWatch, Prometheus, Datadog, Grafana) and log analysis tools.

Key Responsibilities:

• Monitor and support big data operations and batch processing in AWS for Market Regulation; detect, remediate, and escalate processing failures.
• Run and troubleshoot SQL/Postgres/Oracle queries to validate data processing status and integrity.
• Automate data monitoring, alerting, and remediation tasks using Python, Bash, and PowerShell scripts.
• Implement and manage job schedules using JAMS, Broadcom AutoSys, or equivalent tools; resolve scheduling conflicts and failed jobs.
• Maintain hybrid infrastructure across enterprise Linux and Windows systems in on-premises and cloud environments.
• Operate cloud resources in AWS and Azure, manage containers, and support server and application deployments.
• Support CI/CD pipelines (Jenkins, GitLab), identify and resolve deployment failures, and follow git best practices.
• Implement and maintain monitoring and observability for systems, containers, and cloud resources; analyze logs and performance metrics to resolve bottlenecks.
• Document runbooks, test plans, and automation scripts; participate in DR/BCP and SOX testing and reporting.
• Lead or co-lead process improvement initiatives focused on automation, containerization, and AI adoption; collaborate with DevOps, Security, and Application teams.

Education Qualification:

• Bachelors degree in Computer Science, Information Systems, or related field required, or equivalent experience.
• Certifications in AWS, Linux administration, or workload automation are a plus.