Lead Consultant - Data Engineer - AWS

Design and develop data pipelines for Generative AI projects by leveraging a combination of technologies, including Vector DB, Graph DB, Airflow, Spark, PySpark, Python, LangChain, AWS Functions, Redshift, and SSIS. This will involve the logical and efficient integration of these tools to create seamless, high-performance data flows that efficiently support the data requirements of our cutting-edge AI initiatives. Collaborate with data scientists, AI researchers, and other stakeholders to understand data requirements and translate them into effective data engineering solutions. 

User will be managing movement, organization and quality assessments of large set of data to facilitate the creation of Knowledge base for RAG systems and model training 

Demonstrate familiarity with data integration services such as AWS Glue and Azure Data Factory, showcasing the ability to effectively utilize these platforms for seamless data ingestion, transformation, and orchestration across various sources and destinations. 

Possess proficiency in constructing data warehouses and data lakes, demonstrating a strong foundation in organizing and consolidating large volumes of structured and unstructured data for efficient storage, retrieval, and analysis. 

Optimize and maintain data pipelines to ensure high-performance, reliable, and scalable data processing. 

Develop and implement data validation and quality assurance procedures to ensure the accuracy and consistency of the data used in Generative AI projects. 

Monitor and troubleshoot data pipeline performance, identify bottlenecks, and implement improvements as necessary. 

Stay current with emerging trends and technologies in the fields of data engineering, Generative AI, and related areas to ensure the continued success of our projects. 

Collaborate with team members on documentation, knowledge sharing, and best practices for data engineering within a Generative AI context.