Role: Data Engineer
Experience: 7 to 9 years
Skills: Databricks SQL, Databricks Delta Lake, Databricks Workflows, PySpark, Amazon S3, and Amazon Redshift
Responsibility:
- Design and implement data pipelines, ETL processes, and data storage solutions that support data-intensive applications(4+ Years)
- Develop, test, and maintain architectures such as databases and large-scale data processing systems using tools such as Spark, Databricks and SQL (4+ Years)
- Solid in Java/Python, Datastructure and Algorithms (4+)
- Experience designing and delivering Transactional, Warehouse, Analytical and Reporting Data Platforms leveraging modern cloud data technologies (3+ years)
- Experience in cloud development with the AWS platform (3+ years)
- Experience in an Agile/Scrum team environment (2+ years)
- Develop and maintain data pipelines using PySpark to ensure efficient data processing.
- Implement and manage data storage solutions using Amazon S3 and Amazon Redshift.
- Utilize Databricks SQL to perform complex data queries and generate actionable insights.
- Oversee the integration of Databricks Delta Lake for optimized data storage and retrieval.
- Design and automate workflows using Databricks Workflows to streamline data operations.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Ensure data quality and integrity through rigorous testing and validation processes.
- Provide technical support and troubleshooting for data-related issues.
- Lead the development of data models and schemas to support business analytics.
- Conduct performance tuning and optimization of data processing tasks.
- Stay updated with the latest industry trends and technologies to enhance data solutions.
- Document all development processes and maintain comprehensive technical documentation.
- Mentor junior developers and provide guidance on best practices.