Data Engineering Lead - SQL and PySpark

Optum India; Dadri; On-site; INR 1,400,000 - 2,400,000; Full time

Job summary

Optum India is seeking a senior Data Engineer to design scalable data pipelines and lead IDM data integration testing using AI skills. The role requires strong SQL, Python, Spark, and ETL/ELT development, plus hands-on Azure Data Factory and Databricks experience.

Ideal candidates will have 7+ years in data engineering, expertise in gen AI tools, and a track record of improving data quality and governance across enterprise-scale pipelines.

Qualifications

• Bachelor's degree or equivalent experience.
• 7+ years of experience in Data Engineering.
• Solid SQL, Python, Spark, and ETL/ELT development experience.
• Hands-on experience with Azure Data Factory, Databricks, or equivalent cloud platforms.
• Hands on experience with Generative AI technologies including OpenAI, Claude, Gemini, LangChain, Agents, RAG, Vector Databases, Prompt Engineering, and fine-tuning.
• Experience with data governance, data quality, and metadata management frameworks.
• Knowledge of Azure Data Factory, Databricks, SQL, Spark, Gen AI, or equivalent cloud-native data platforms.

Responsibilities

• Design and implement scalable data pipelines.
• Develop IDM data integration testing solutions using AI Skill sets.
• Perform data validation, cleansing, reconciliation, and governance frameworks to ensure accuracy and consistency of identity-related data.
• Optimize ETL/ELT processes, automate data workflows, and improve pipeline reliability, scalability, and monitoring.
• Work closely with IDM architects, security teams, application owners, and business stakeholders to define data requirements and data validation rules.
• Analyzes and investigates.
• Provides explanations and interpretations within area of expertise.
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements.

Skills

SQL

Python

Spark

ETL/ELT

Generative AI

Data governance

SQL optimization

Education

Bachelor's degree or equivalent experience

Tools

Azure Data Factory

Databricks

LangChain

Vector Databases

Prompt Engineering

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Primary Responsibilities

• Design and implement scalable data pipelines
• Develop IDM data integration testing solutions using AI Skill sets
• Perform data validation, cleansing, reconciliation, and governance frameworks to ensure accuracy and consistency of identity-related data
• Optimize ETL/ELT processes, automate data workflows, and improve pipeline reliability, scalability, and monitoring
• Work closely with IDM architects, security teams, application owners, and business stakeholders to define data requirements and data validation rules
• Analyzes and investigates
• Provides explanations and interpretations within area of expertise
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications

• Bachelor's degree or equivalent experience
• 7+ years of experience in Data Engineering
• Solid SQL, Python, Spark, and ETL/ELT development experience
• Hands-on experience with Azure Data Factory, Databricks, or equivalent cloud platforms
• Hands on experience required in Generative AI technologies including OpenAI, Claude, Gemini, LangChain, Agents, RAG, Vector Databases, Prompt Engineering, and fine-tuning
• Experience with data governance, data quality, and metadata management frameworks
• Knowledgeof Azure Data Factory, Databricks, SQL, Spark, Gen AI, or equivalent cloud-native data platforms

Preferred Qualifications

• Proven excellent communication and teamwork skills
• Proven great attention to detail
• Proven organizational skills
• Proven analytical mind

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.