Data Engineering Lead - SQL and PySpark

Optum; Dadri; On-site; INR 1,400,000 - 2,400,000; Full time

Job summary

Optum is hiring a senior data engineer to design scalable data pipelines and optimize ETL/ELT workflows. You will work with Azure Data Factory and Databricks, applying Generative AI techniques to improve data quality and governance.

The role involves partnering with security, IDM architects, and stakeholders to define data requirements and ensure reliable data delivery. The ideal candidate has 7+ years in data engineering, strong SQL/Python/Spark skills, and hands-on AI tool experience.

Qualifications

• Bachelor's degree or equivalent experience.
• 7+ years of experience in Data Engineering.
• Solid SQL, Python, Spark, and ETL/ELT development experience.
• Hands-on with Azure Data Factory, Databricks, or equivalent cloud platforms.
• Hands on experience with Generative AI technologies including OpenAI, Claude, Gemini, LangChain, Agents, RAG, Vector Databases, Prompt Engineering, and fine-tuning.
• Experience with data governance, data quality, and metadata management.
• Knowledge of Azure Data Factory, Databricks, SQL, Spark, Gen AI, or equivalent cloud-native data platforms.

Responsibilities

• Design and implement scalable data pipelines.
• Develop IDM data integration testing solutions using AI Skill sets.
• Perform data validation, cleansing, reconciliation, and governance frameworks.
• Optimize ETL/ELT processes, automate data workflows, and improve pipeline reliability.
• Collaborate with IDM architects, security teams, and business stakeholders to define data requirements and rules.

Skills

SQL

Python

Spark

ETL/ELT

Generative AI

Data governance

Data quality

Metadata mgmt

Communication

Education

Bachelor's degree

Tools

Azure Data Factory

Databricks

SQL

Spark

Gen AI platforms

OpenAI

Claude

Gemini

LangChain

RAG

Vector Databases

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Functions may include database architecture, engineering, design, optimization, security, and administration; as well as data modeling, big data development, Extract, Transform, and Load (ETL) development, storage engineering, data warehousing, data provisioning and other similar roles. Responsibilities may include Platform-as-a-Service and Cloud solution with a focus on data stores and associated eco systems. Duties may include management of design services, providing sizing and configuration assistance, ensuring strict data quality, and performing needs assessments. Analyzes current business practices, processes and procedures as well as identifying future business opportunities for leveraging data storage and retrieval system capabilities. Manages relationships with software and hardware vendors to understand the potential architectural impact of different vendor strategies and data acquisition. May design schemas, write SQL or other data markup scripting and helps to support development of Analytics and Applications that build on top of data. Selects, develops and evaluates personnel to ensure the efficient operation of the function.

Primary Responsibilities:

• Design and implement scalable data pipelines
• Develop IDM data integration testing solutions using AI Skill sets
• Perform data validation, cleansing, reconciliation, and governance frameworks to ensure accuracy and consistency of identity-related data
• Optimize ETL/ELT processes, automate data workflows, and improve pipeline reliability, scalability, and monitoring
• Work closely with IDM architects, security teams, application owners, and business stakeholders to define data requirements and data validation rules
• Analyzes and investigates
• Provides explanations and interpretations within area of expertise
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
  

Required Qualifications:

• Bachelor's degree or equivalent experience
• 7+ years of experience in Data Engineering
• Solid SQL, Python, Spark, and ETL/ELT development experience
• Hands-on experience with Azure Data Factory, Databricks, or equivalent cloud platforms
• Hands on experience required in Generative AI technologies including OpenAI, Claude, Gemini, LangChain, Agents, RAG, Vector Databases, Prompt Engineering, and fine-tuning
• Experience with data governance, data quality, and metadata management frameworks
• Knowledge of Azure Data Factory, Databricks, SQL, Spark, Gen AI, or equivalent cloud-native data platforms

Preferred Qualifications:

• Proven excellent communication and teamwork skills
• Proven great attention to detail
• Proven organizational skills
• Proven analytical mind

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.