Data Engineering Manager - Azure, GenAI, RAG, Agentic AI

Optum; Dadri; On-site; INR 3,500,000 - 5,500,000; Full time

Job summary

Optum is a global health services company seeking a skilled Data/ML Engineer to own the end-to-end ML lifecycle, from data ingestion to deployment and governance. You will architect and operate enterprise-scale ML pipelines on Azure, leveraging IaC, CI/CD, MLOps, and advanced tooling.

The role requires strong hands-on experience with Azure Databricks, Spark, Delta Lake, Kafka, and OpenAI/Azure AI components, plus Python, SQL and PySpark.

Qualifications

• Bachelor's or graduate degree in a related field or equivalent experience.
• 6+ years of experience in Data Engineering.
• Hands-on expertise with Azure Databricks, Spark, Delta Lake and streaming technologies.

Responsibilities

• Own end-to-end ML lifecycle from data ingestion to deployment and governance.
• Build and run enterprise-grade ML pipelines on Azure with IaC and CI/CD.
• Manage Azure ML and MLflow for experiment tracking and model registry.
• Develop Agentic AI workflows with multi-step reasoning and guardrails.
• Design scalable batch/real-time data pipelines using Azure Databricks, Spark, Delta Lake, streaming tech.
• Handle structured, semi-structured, and unstructured data sources.

Skills

Data modeling

Team leadership

Problem solving

Education

Graduate degree or equivalent experience

Tools

Azure Databricks

Spark

Delta Lake

Kafka

Event Hubs

Azure Machine Learning

MLflow

MLOps

CI/CD

GitHub Actions

Azure DevOps

Terraform

Bicep

Python

SQL

PySpark

Azure OpenAI

Pinecone

Vector Databases

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Primary Responsibilities:

• Own and drive the end-to-end Machine Learning lifecycle, including data ingestion, feature engineering, model development, training, evaluation, deployment, monitoring, retraining, governance, and rollback strategies
• Architect, build, and operate enterprise-scale ML platforms and production-grade pipelines on Azure, leveraging Infrastructure-as-Code, CI/CD, MLOps, and automation best practices
• Implement and manage Azure Machine Learning and MLflow for experiment tracking, model versioning, model registry, reproducibility, and controlled promotion across Development, Test, and Production environments
• Engineer and operationalize Agentic AI workflows incorporating multi-step reasoning, tool integrations, workflow orchestration, guardrails, observability, reliability, security, and cost optimization
• Design and build scalable batch and real-time data processing pipelines using Azure Databricks, Spark, Delta Lake, and streaming technologies
• Develop robust data pipelines for structured, semi-structured, and unstructured data, including documents, text, logs, images, and other multi-modal data sources
• Build and optimize RAG pipelines, including document ingestion, chunking strategies, embedding generation, indexing, retrieval optimization, re-ranking, and response evaluation frameworks
• Develop high-performance batch and streaming ingestion frameworks utilizing Spark, Kafka, Event Hubs, and cloud-native messaging services
• Establish data quality, lineage, monitoring, and governance frameworks to ensure reliability, compliance, and operational excellence across AI/ML solutions
• Collaborate with data scientists, ML engineers, architects, and business stakeholders to translate business requirements into scalable AI and data platform solutions
• Mentor junior engineers and provide technical leadership on architecture, engineering standards, best practices, and enterprise AI adoption initiatives
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Graduate degree or equivalent experience
• 6+ years of experience in Data Engineering
• Solid hands-on experience with:
  • Azure Databricks, Spark, Delta Lake, Kafka, Event Hubs
  • Azure Machine Learning, MLflow, MLOps, CI/CD
  • Azure OpenAI, Generative AI, RAG, Agentic AI
  • Azure AI Search, Pinecone, Vector Databases
  • Python, SQL, PySpark
  • GitHub Actions, Azure DevOps, Terraform/Bicep
  • Distributed Data Processing and Real-Time Data Streaming
  • Enterprise Data Architecture, Data Governance, and Cloud-Native Engineering

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.