Job Description
:Key Responsibilities:
Data Engineering & Architecture:
Design, develop, and maintainhigh-performance data pipelinesfor structured and unstructured data usingAzure Data BricksandApache Spark. Build and manage scalabledata ingestion frameworksfor batch and real-time data processing. Implement and optimizedata lake architectureinAzure Data Laketo support analytics and reporting workloads. Develop and optimizedata modelsand queries inAzure Synapse Analyticsto power BI and analytics use cases. Cloud-Based Data Solutions:
Architect and implement moderndata lakehousescombining the best of data lakes and data warehouses. LeverageAzure serviceslikeData Factory,Event Hub, andBlob Storagefor end-to-end data workflows. Ensure security, compliance, and governance of data throughAzure Role-Based Access Control (RBAC)andData Lake ACLs. ETL/ELT Development:
Develop robustETL/ELT pipelinesusingAzure Data Factory,Data Bricks notebooks, andPySpark. Perform data transformations, cleansing, and validation to prepare datasets for analysis. Manage and monitor job orchestration, ensuring pipelines run efficiently and reliably. Performance Optimization:
OptimizeSpark jobsandSQL queriesfor large-scale data processing. Implementpartitioning, caching, andindexing strategiesto improve performance and scalability of big data workloads. Conductcapacity planningand recommend infrastructure optimizations for cost-effectiveness. Collaboration & Stakeholder Management:
Work closely with business analysts, data scientists, and product teams to understand data requirements and deliver solutions. Participate incross-functional design sessionsto translate business needs into technical specifications. Provide thought leadership onbest practicesin data engineering and cloud computing. Documentation & Knowledge Sharing:
Create detailed documentation for data workflows, pipelines, and architectural decisions. Mentor junior team members and promote a culture of learning and innovation. Required Qualifications:
Experience:7+ years of experiencein data engineering, big data, or cloud-based data solutions.Proven expertise withAzure Data Bricks,Azure Data Lake, andAzure Synapse Analytics. Technical Skills:Strong hands-on experience withApache Sparkand distributed data processing frameworks.Advanced proficiency inPythonandSQLfor data manipulation and pipeline development.Deep understanding ofdata modelingfor OLAP, OLTP, and dimensional data models.Experience withETL/ELT toolslike Azure Data Factory or Informatica.Familiarity withAzure DevOpsfor CI/CD pipelines and version control. Big Data Ecosystem:Familiarity withDelta Lakefor managing big data in Azure.Experience withstreaming data frameworkslike Kafka, Event Hub, or Spark Streaming. Cloud Expertise:Strong understanding ofAzure cloud architecture, includingstorage, compute, and networking.Knowledge ofAzure security best practices, such as encryption and key management. Preferred Skills (Nice to Have):
Experience withmachine learning pipelinesand frameworks like MLFlow or Azure Machine Learning. Knowledge ofdata visualization toolssuch as Power BI for creating dashboards and reports. Familiarity withTerraformorARM templatesfor infrastructure as code (IaC). Exposure toNoSQL databaseslike Cosmos DB or MongoDB. Experience withdata governanceto
Weekly Hours:
40Time Type:
RegularLocation:
Hyderabad, Andhra Pradesh, IndiaIt is the policy of AT&T to provide equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. In addition, AT&T will provide reasonable accommodations for qualified individuals with disabilities.AT&T is a fair chance employer and does not initiate a background check until an offer is made.
