Senior Data Engineering Lead

Optum; Pune District, Chennai District; Hybrid; INR 1,800,000 - 2,600,000; Full time

Job summary

Optum in Pune, India is seeking an experienced Data Engineer with 5+ years in data engineering to design and optimize scalable data pipelines using Spark, PySpark, and Databricks.

The role requires deep knowledge of AWS services (S3, IAM, VPC) and modern data stack elements like Delta Lake, Unity Catalog, and Iceberg, as well as CI/CD, Docker, and Terraform.

You will work with healthcare data concepts (FHIR/HL7, PHI) and implement reliable ingestion with quality controls and replay capabilities.

Qualifications

• 5+ years of experience in data engineering.
• Advanced Python, SQL, and Spark/PySpark.
• Production Databricks experience with Delta Lake, Workflows, Unity Catalog, notebooks/jobs, SQL Warehouses, and cluster optimization.
• Hands-on AWS experience with S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS.
• Apache Iceberg knowledge including tables, catalogs, snapshots, schema evolution, compaction and interoperability with query engines.
• Experience with Git, pull requests, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and incident response.
• Experience processing semi-structured data and building resilient ingestion pipelines with quality controls, error handling, and replay capability.
• Healthcare data knowledge: FHIR R4 and HL7 v2, OMOP CDM, clinical terminologies, PHI, HIPAA-aligned engineering controls, and de-identification concepts.
• Demonstrated responsible use of AI coding tools and the ability to critically review, test, and productionize generated code.

Skills

Data Engineering

Python

SQL

Spark

PySpark

Databricks

AWS

S3

IAM

VPC

Terraform

CI/CD

Docker

Iceberg

Tools

Databricks

Delta Lake

Workflows

Unity Catalog

Notebooks/Jobs

SQL Warehouses

Cluster Optimization

Git

CI/CD

Job description

• Overall 5+ YRS of experience in Data Engineering
• Advanced Python, SQL, and Apache Spark/PySpark; strong understanding of distributed processing and performance tuning.
• Production Databricks experience with Auto Loader, Delta Lake, Workflows, Unity Catalog, notebooks/jobs, SQL Warehouses, and cluster/job optimization.
• Hands-on AWS experience with S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS.
• Practical Apache Iceberg knowledge, including tables, catalogs, snapshots, schema/partition evolution, compaction, and interoperability with query engines.
• Experience with Git, pull requests, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and incident response.
• Experience processing semi-structured data and building resilient ingestion pipelines with quality controls, error handling, and replay capability.
• Healthcare data knowledge: FHIR R4 and/or HL7 v2, OMOP CDM, clinical terminologies, PHI, HIPAA-aligned engineering controls, and de-identification concepts.
• Demonstrated responsible use of AI coding tools and the ability to critically review, test, and productionize generated code.