Data Engineering Lead - SQL and PySpark

UnitedHealth Group; India; On-site; INR 2,000,000 - 4,500,000; Full time

Job summary

UnitedHealth Group's Optum is seeking an experienced Data Engineer to design scalable data pipelines and manage end-to-end data integration tasks in a cloud-first environment. You will collaborate with IDM architects, security, and business stakeholders to ensure data quality and governance across platforms.

The role emphasizes hands-on GenAI capabilities, data validation, and process optimization to support health data initiatives and analytics at scale.

Qualifications

• Bachelor's degree or equivalent experience.
• 7+ years of Data Engineering experience.
• Solid SQL, Python, Spark, and ETL/ELT development experience.
• Hands-on experience with Azure Data Factory, Databricks, or equivalent cloud platforms.
• Hands on experience with Generative AI technologies including OpenAI, Claude, Gemini, LangChain, Agents, RAG, Vector Databases, Prompt Engineering, and fine-tuning.
• Experience with data governance, data quality, and metadata management frameworks.
• Knowledge of Azure Data Factory, Databricks, SQL, Spark, Gen AI, or equivalent cloud-native data platforms.

Responsibilities

• Design and implement scalable data pipelines
• Develop IDM data integration testing solutions using AI Skill sets
• Perform data validation, cleansing, reconciliation, and governance frameworks to ensure accuracy and consistency of identity-related data
• Optimize ETL/ELT processes, automate data workflows, and improve pipeline reliability, scalability, and monitoring
• Work closely with IDM architects, security teams, application owners, and business stakeholders to define data requirements and data validation rules
• Analyzes and investigates
• Provides explanations and interpretations within area of expertise
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Skills

SQL

Python

Spark

ETL/ELT

AI/GenAI

Education

Bachelor's degree or equivalent experience

Tools

Azure Data Factory

Databricks

LangChain

Vector Databases

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Functions may include database architecture, engineering, design, optimization, security, and administration; as well as data modeling, big data development, Extract, Transform, and Load (ETL) development, storage engineering, data warehousing, data provisioning and other similar roles. Responsibilities may include Platform-as-a-Service and Cloud solution with a focus on data stores and associated eco systems. Duties may include management of design services, providing sizing and configuration assistance, ensuring strict data quality, and performing needs assessments. Analyzes current business practices, processes and procedures as well as identifying future business opportunities for leveraging data storage and retrieval system capabilities. Manages relationships with software and hardware vendors to understand the potential architectural impact of different vendor strategies and data acquisition. May design schemas, write SQL or other data markup scripting and helps to support development of Analytics and Applications that build on top of data. Selects, develops and evaluates personnel to ensure the efficient operation of the function.

Primary Responsibilities:

• Design and implement scalable data pipelines
• Develop IDM data integration testing solutions using AI Skill sets
• Perform data validation, cleansing, reconciliation, and governance frameworks to ensure accuracy and consistency of identity-related data
• Optimize ETL/ELT processes, automate data workflows, and improve pipeline reliability, scalability, and monitoring
• Work closely with IDM architects, security teams, application owners, and business stakeholders to define data requirements and data validation rules
• Analyzes and investigates
• Provides explanations and interpretations within area of expertise
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
  

Required Qualifications:

• Bachelor's degree or equivalent experience
• 7+ years of experience in Data Engineering
• Solid SQL, Python, Spark, and ETL/ELT development experience
• Hands-on experience with Azure Data Factory, Databricks, or equivalent cloud platforms
• Hands on experience required in Generative AI technologies including OpenAI, Claude, Gemini, LangChain, Agents, RAG, Vector Databases, Prompt Engineering, and fine-tuning
• Experience with data governance, data quality, and metadata management frameworks
• Knowledge of Azure Data Factory, Databricks, SQL, Spark, Gen AI, or equivalent cloud-native data platforms

Preferred Qualifications:

• Proven excellent communication and teamwork skills
• Proven great attention to detail
• Proven organizational skills
• Proven analytical mind

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.