Senior Data Engineering Lead

UnitedHealth Group; Chennai District; On-site; INR 2,000,000 - 3,500,000; Full time

Job summary

UnitedHealth Group seeks a senior data engineer to build reliable data pipelines on AWS and Databricks, leveraging Delta Lake and Iceberg for scalable storage and governance.

You will develop ingestion frameworks for healthcare data (FHIR/HL7/OMOP CDM) with strong data quality and privacy controls, and implement CI/CD, tests, and infrastructure as code across cloud services.

Qualifications

• 4+ years of production data-engineering experience.

Responsibilities

• Build reliable batch, micro-batch, and event-driven pipelines on AWS and Databricks.
• Develop reusable ingestion frameworks for REST APIs, FHIR Bulk Export, HL7 interfaces, databases, SFTP/file exchange, JSON/NDJSON, CSV, XML, PDFs, and clinical text.
• Implement scalable Spark/PySpark and SQL transformations, including schema inference/evolution, checkpointing, idempotency, retries, backfills, and replay.
• Design and maintain Delta Lake and Apache Iceberg tables, including physical design, partitioning/clustering, compaction, file sizing, incremental reads/writes, performance tuning, and cost optimization.
• Build curated healthcare data models and transformations using FHIR, HL7, OMOP CDM, and clinical terminology mappings.
• Implement data-quality frameworks: schema validation, referential-integrity checks, business-rule testing, anomaly detection, reconciliation, completeness checks, and data-quality observability.
• Build metadata and lineage capture across source systems, pipeline runs, code versions, transformation rules, mappings, and published data products.
• Implement privacy-aware data processing for PHI, including access controls, masking, tokenization/pseudonymization, de-identification, and auditable handling patterns.
• Deliver CI/CD pipelines, automated unit/integration/data tests, Terraform or CloudFormation, containerized services, monitoring, alerting, runbooks, and incident-recovery procedures.
• Integrate and optimize governed data consumption through Databricks SQL, Athena, Snowflake, Trino, or equivalent engines.
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Skills

Databricks

AWS

Python

SQL

Spark

CI/CD

Git

Docker

Terraform

Unit tests

Education

Undergraduate degree

Tools

Delta Lake

Iceberg

Unity Catalog

S3

Terraform

CloudFormation

KMS

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Primary Responsibilities:

• Build reliable batch, micro-batch, and event-driven pipelines on AWS and Databricks
• Develop reusable ingestion frameworks for REST APIs, FHIR Bulk Export, HL7 interfaces, databases, SFTP/file exchange, JSON/NDJSON, CSV, XML, PDFs, and clinical text
• Implement scalable Spark/PySpark and SQL transformations, including schema inference/evolution, checkpointing, idempotency, retries, backfills, and replay
• Design and maintain Delta Lake and Apache Iceberg tables, including physical design, partitioning/clustering, compaction, file sizing, incremental reads/writes, performance tuning, and cost optimization
• Build curated healthcare data models and transformations using FHIR, HL7, OMOP CDM, and clinical terminology mappings
• Implement data-quality frameworks: schema validation, referential-integrity checks, business-rule testing, anomaly detection, reconciliation, completeness checks, and data-quality observability
• Build metadata and lineage capture across source systems, pipeline runs, code versions, transformation rules, mappings, and published data products
• Implement privacy-aware data processing for PHI, including access controls, masking, tokenization/pseudonymization, de-identification, and auditable handling patterns
• Deliver CI/CD pipelines, automated unit/integration/data tests, Terraform or CloudFormation, containerized services, monitoring, alerting, runbooks, and incident-recovery procedures

• Integrate and optimize governed data consumption through Databricks SQL, Athena, Snowflake, Trino, or equivalent engines

• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications:

• Undergraduate degree or equivalent experience
• 4+ years of production data-engineering experience
• Production Databricks experience with Auto Loader, Delta Lake, Workflows, Unity Catalog, notebooks/jobs, SQL Warehouses, and cluster/job optimization
• Hands-on AWS experience with S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS
• Experience with Git, pull requests, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and incident response
• Experience processing semi-structured data and building resilient ingestion pipelines with quality controls, error handling, and replay capability
• Practical Apache Iceberg knowledge, including tables, catalogs, snapshots, schema/partition evolution, compaction, and interoperability with query engines
• Healthcare data knowledge: FHIR R4 and/or HL7 v2, OMOP CDM, clinical terminologies, PHI, HIPAA-aligned engineering controls, and de-identification concepts
• Advanced Python, SQL, and Apache Spark/PySpark; solid understanding of distributed processing and performance tuning
• Demonstrated responsible use of AI coding tools and the ability to critically review, test, and productionize generated code.

Preferred Qualification:

• Healthcare or regulated-data experience

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.