Skill: Databricks
Experience: 6 to 13 years
Location: AIA Group-Bhubaneswar
Job description
Responsibilities:
- Designing, building, and deploying data pipelines and workflows using Databricks.
- Developing and optimizing Apache Spark applications for data processing and analytics.
- Integrating Databricks with various data sources and platforms.
- Implementing data quality checks and monitoring processes.
- Managing and configuring the Databricks environment, including clusters and notebooks.
- Troubleshooting and resolving issues related to Databricks performance and functionality.
- Collaborating with data scientists and analysts to implement machine learning models.
- Staying up-to-date with the latest Databricks features and best practices.
Qualifications:
- Strong proficiency in Python or Scala programming languages.
- Experience with Apache Spark and big data processing frameworks.
- Knowledge of cloud platforms like AWS, Azure, or GCP.
- Familiarity with data warehousing and ETL processes.
- Understanding of machine learning concepts and algorithms.
- Strong problem-solving and analytical skills.
Additional Skills (depending on the specific role):
- Experience with Databricks Delta Lake or MLflow.
- Knowledge of infrastructure as code (IaC) tools like Terraform.
- Familiarity with CI/CD pipelines for data and machine learning workflows.
- Experience with containerization technologies like Docker.