Specialist Data/AI Engineering (Azure Databricks platform engineer)

Job Description:

Roles & Responsibilities

• Minimum 6+ years of experience in the fundamental technology work of architecting/designing/building/managing/Optimizing the core frameworks for our cloud-based system platforms and end-to-end solutions.
• Responsible for optimization of cloud infrastructure, including Databricks, ADLS, Blob, VMs etc.,
• Conduct Proof-of-Concept activities in support of advanced use cases.
• Must have architectural focus for ensuring availability, reliability and scalability are necessary
• Responsible for documenting the technical solutions
• Administer and manage Azure Databricks workspaces, clusters, and jobs with a focus on maximizing efficiency and performance.
• Implement and manage Unity Catalog to provide centralized governance and fine-grained access control across data assets.
• Design and execute cost optimization strategies, including monitoring cluster usage, autoscaling, and resource allocation to minimize cloud expenses.
• Collaborate with data engineers and security teams to enforce data governance, compliance, and security policies using Unity Catalog.
• Automate deployment, scaling, and monitoring tasks using scripting and cloud-native tools.
• Troubleshoot and resolve issues related to Databricks clusters, jobs, and Unity Catalog configurations.
• Responsible for working with international teams spread across multiple time zones.

Must Have:

• 6 to 8 years of hands-on experience administering Azure Databricks in cloud environments.
• In-depth knowledge and experience with Unity Catalog for data governance and security.
• Proven expertise in cost optimization for Databricks clusters and cloud resources.
• Strong understanding of Databricks architecture, cluster management, and job orchestration.
• experience in RDBMS, ANSI SQL/PL SQL, Hadoop, Python and/or Java,spark/Pyspark.
• Proficiency in Spark architecture, including cluster management, job scheduling, and performance tuning.
• Experience in  Azure DevOps, Git/Github.
• Azure Event Hub/ Azure Event Grid / Kafka
• Hands-on experience with scripting languages such as Python, Scala, or SQL.
• Experience with infrastructure-as-code tools like Terraform or ARM templates.
• Solid understanding of data security, governance, and compliance standards.
• Excellent problem-solving skills and ability to work in a collaborative environment.
• Strong communication skills and ability to explain technical concepts to non-technical stakeholders.

Additional experience:

• Hadoop Administration and/or Azure cloud Administration 
• Working with Object storage ( ADLS/Blob or S3), Kubernetes (preferably AKS)
• Azure AD/ OpenID /OAuth/ SAML
• NSG /Firewall deployment and usage
• DNS configuration and management
• Automation through Ansible or Teraform or Azure DevOps
• Basic Linux administration and Shell scripting (Bash)
• Good documentation skills
• Ability to troubleshoot to isolate and diagnose common technical problems
• Monitoring and use of monitoring tools for managing critical infrastructure

Preferred Certifications:

• Microsoft Certified: Azure Administrator Associate/ Microsoft Certified: Azure Solutions Architect Expert
• Databricks Accredited Platform Administrator Accreditation

Weekly Hours:

Time Type:

Location:

It is the policy of AT&T to provide equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. In addition, AT&T will provide reasonable accommodations for qualified individuals with disabilities. AT&T is a fair chance employer and does not initiate a background check until an offer is made.