Consultant-Data Engineer, AWS+Python, Spark, Kafka for ETL

Develop, deploy, and manage ETL pipelines using AWS services, Python, Spark, and Kafka.

Integrate structured and unstructured data from various data sources into data lakes and data warehouses.

Design and deploy scalable, highly available, and fault-tolerant AWS data processes using AWS data services (Glue, Lambda, Step, Redshift)

Monitor and optimize the performance of cloud resources to ensure efficient utilization and cost-effectiveness.

Implement and maintain security measures to protect data and systems within the AWS environment, including IAM policies, security groups, and encryption mechanisms.

Migrate the application data from legacy databases to Cloud based solutions (Redshift, DynamoDB, etc) for high availability with low cost

Develop application programs using Big Data technologies like Apache Hadoop, Apache Spark, etc with appropriate cloud-based services like Amazon AWS, etc.

Build data pipelines by building ETL processes (Extract-Transform-Load)

Implement backup, disaster recovery, and business continuity strategies for cloud-based applications and data.

Responsible for analysing business and functional requirements which involves a review of existing system configurations and operating methodologies as well as understanding evolving business needs