Data Engineering Lead - Python, Pyspark, Azure Databricks

UnitedHealth Group; Gurugram District; On-site; INR 2,000,000 - 3,200,000; Full time

Job summary

Optum, part of UnitedHealth Group, seeks a senior Data Engineer to design, build and optimize scalable batch and streaming data pipelines on Azure and Databricks. You will ingest diverse data formats, implement governance, and partner with architects, analysts and data scientists to deliver trusted insights.

You bring 6+ years in data engineering, strong PySpark/SQL skills, Delta Lake and Unity Catalog, and hands-on Azure services (ADF, ADLS Gen2, Key Vault, Azure SQL, Power BI).

Qualifications

• B.E./B.Tech./MCA or equivalent in CS/IT/Engineering.
• 6+ years data engineering with hands-on Azure Databricks.
• Delta Lake, Lakehouse, Unity Catalog experience.
• File formats: CSV, JSON, XML, Parquet, Avro, Delta; schema evolution.
• Proficient in Python, PySpark and SQL; Spark internals knowledge.
• Strong problem-solving and communication skills.
• Azure services: ADF, ADLS Gen2, Key Vault, Azure SQL, Power BI.

Responsibilities

• Design and maintain scalable batch and streaming data pipelines on Azure and Databricks.
• Ingest and process structured, semi-structured and unstructured data formats.
• Develop transformations in PySpark and Spark SQL with CDC/SCD handling.
• Design data models and consumption objects (tables, views, materialized views).
• Implement data governance and security using Unity Catalog (PHI/PII protection).
• Build data quality, validation and reconciliation frameworks.
• Tune Spark jobs and Delta tables for performance and cost.
• Drive CI/CD and DataOps with Azure DevOps or GitHub.

Skills

Python

PySpark

SQL

Azure Databricks

Delta Lake

Unity Catalog

Spark tuning

CI/CD

Git

Data modeling

Education

B.E./B.Tech./MCA or equivalent

Tools

Azure Data Factory

ADLS Gen2

Key Vault

Power BI

Databricks

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Primary Responsibilities:

• Design, build and maintain scalable batch and streaming data pipelines on Azure and Databricks using the Medallion Architecture (Bronze, Silver, Gold) on Delta Lake
• Ingest and process structured, semi-structured and unstructured data in formats such as CSV, JSON, XML, Parquet, Avro, ORC, Excel and Delta, using Azure Data Factory, Auto Loader and Lakeflow Spark Declarative Pipelines
• Develop complex transformations in PySpark and Spark SQL, including CDC, SCD Type 1/2, deduplication and handling of nested or schema-evolving data
• Design data models and build the right consumption objects for each use case: managed/external tables, views, temporary views, materialized views and streaming tables
• Implement data governance, lineage and security using Unity Catalog, including row/column-level security and PHI/PII protection
• Build data quality, validation and reconciliation frameworks to keep data accurate and reliable
• Tune Spark jobs and Delta tables for performance and cost (partitioning, liquid clustering, Z-ordering, OPTIMIZE/VACUUM, compute sizing)
• Drive CI/CD and DataOps practices using Azure DevOps or GitHub, Declarative Automation Bundles and automated testing
• Troubleshoot production issues, do root cause analysis and put preventive fixes in place
• Work with architects, analysts, data scientists and business stakeholders, and mentor junior engineers
• Use AI-assisted engineering tools to improve development, testing and documentation
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications:

• B.E./B.Tech./MCA or equivalent degree in Computer Science, IT or Engineering
• 6+ years of data engineering experience, including 3+ years hands-on with Azure Databricks
• Hands-on experience with Delta Lake, Lakehouse architecture and Unity Catalog
• Experience working with file formats such as CSV, JSON, XML, Parquet, Avro and Delta, including schema inference, schema evolution and nested data handling
• Experience with Azure services: ADF, ADLS Gen2, Key Vault, Azure SQL , Power Bi
• Solid knowledge of data modeling, ETL/ELT design, Git, CI/CD and Agile delivery
• Good understanding of views, temporary/global temporary views, materialized views, streaming tables and when to use each
• Solid proficiency in Python, PySpark and SQL, with a solid understanding of Spark internals and performance tuning
• Proven solid problem-solving and communication skills

Preferred Qualifications:

• Databricks Data Engineer Associate/Professional or Microsoft DP-203/DP-700 certification
• Experience with Structured Streaming, Kafka/Event Hubs, Lakeflow Jobs and Databricks SQL
• Infrastructure as Code experience
• Exposure to healthcare data (claims, prior authorization, HL7/FHIR, EDI 837/278) and HIPAA compliance
• Experience with Power BI or AI/BI dashboards
• Experience mentoring or leading a team, or driving cloud migration and modernization

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.