Principal Data Engineer

Optum; Bengaluru; On-site; INR 2,800,000 - 5,200,000; Full time

Job summary

Optum is seeking a Senior Data Platform Engineer in Bengaluru to design and maintain scalable data pipelines for analytics, ML, and GenAI use cases. You will build ingestion frameworks, enable ML workflows, and collaborate with product and analytics teams to deliver high-impact data solutions.

The role emphasizes data quality, observability, and governance, with leadership responsibilities and mentorship across teams. Strong Python/Spark/Databricks and cloud experience are required.

Qualifications

• Bachelor's degree or equivalent experience.
• 5+ years designing, building, and operating production data pipelines and platforms.
• 5+ years of Python (preferred) and/or Java development; code reviews, packaging, deployment.
• 5+ years Spark (PySpark) and Databricks experience.
• 2+ years Generative AI in production environments.
• Cloud experience with AWS/Azure/GCP and secure data handling.
• Experience with data lakes/warehouses (Databricks, Snowflake).
• Experience with ELT/ETL, Airflow, CI/CD, containers, and IaC.

Responsibilities

• Design, develop, and maintain scalable data pipelines and platforms for analytics, ML, and AI use cases.
• Build and optimize ingestion frameworks for large-scale structured and unstructured data.
• Collaborate with cross-functional stakeholders to define long-term technical solutions.
• Enable and support ML/AI workflows, feature engineering, and model deployment.
• Lead initiatives around Generative AI, data quality, observability, lineage, and governance.
• Develop reusable libraries to accelerate delivery across teams.
• Mentor engineers and lead through influence.
• Maintain high ethical standards and confidentiality.

Skills

Python

Spark

Databricks

Airflow

Docker

Kubernetes

GitHub Actions

Terraform

Snowflake

Great Expectations

Deequ

ML/AI workflows

SQL

Data quality

Education

Bachelor's degree or equivalent experience

Tools

Python

Spark

Databricks

Airflow

Docker

Kubernetes

Terraform

GitHub Actions

Snowflake

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Functions may include database architecture, engineering, design, optimization, security, and administration; as well as data modeling, big data development, Extract, Transform, and Load (ETL) development, storage engineering, data warehousing, data provisioning and other similar roles. Responsibilities may include Platform-as-a-Service and Cloud solution with a focus on data stores and associated eco systems. Duties may include management of design services, providing sizing and configuration assistance, ensuring strict data quality, and performing needs assessments. Analyzes current business practices, processes and procedures as well as identifying future business opportunities for leveraging data storage and retrieval system capabilities. Manages relationships with software and hardware vendors to understand the potential architectural impact of different vendor strategies and data acquisition. May design schemas, write SQL or other data markup scripting and helps to support development of Analytics and Applications that build on top of data. Selects, develops and evaluates personnel to ensure the efficient operation of the function.

Primary Responsibilities:

• Design, develop, and maintain scalable data pipelines and data platforms supporting analytics, machine learning, and AI use cases
• Build and optimize ingestion frameworks for large-scale structured and unstructured data, including streaming and event-driven sources
• Partner with cross-functional stakeholders to understand evolving data and AI needs and define long-term technical solutions
• Enable and support machine learning and AI workflows, including feature engineering, data preparation, and model deployment support
• Drive strategic initiatives around Generative AI, data quality, observability, lineage, and governance
• Develop and maintain frameworks that support rapid experimentation and deployment of AI/ML solutions
• Introduce and evolve best practices in data modeling, orchestration, testing, and monitoring
• Identify and champion opportunities for platform scalability, performance optimization, and cost efficiency
• Collaborate with product, analytics, and infrastructure teams to deliver high-impact data and AI solutions
• Build and maintain reusable parsing, enrichment, analytic, and service libraries to accelerate delivery across teams
• Work comfortably under time-sensitive conditions while ensuring thoroughness
• Maintain high ethical standards and the ability to remain objective and confidential
• Reviews the work of others
• Develops innovative approaches
• Sought out as expert
• Serves as a leader/ mentor
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Bachelor's degree or equivalent experience
• 5+ years of experience designing, building, and operating production data pipelines and platforms
• 5+ years of hands-on development with Python (preferred) and/or Java, including code reviews, packaging, and deployment
• 5+ years of experience with Spark (PySpark) and Databricks (or similar distributed data processing platform)
• 2+ years of experience leveraging and deploying Generative AI use cases to production environments
• Cloud experience (AWS, Azure, and/or GCP), including secure handling of sensitive data (PII/PHI) and collaboration with compliance partners
• Experience building and operating production data platforms and pipelines across batch and streaming workloads
• Experience with distributed processing and lakehouse/warehouse patterns (eg, Spark/PySpark, Databricks, Snowflake, Microsoft Fabric)
• Experience building ingestion frameworks for structured and unstructured data, including event/log and semi-structured formats
• Experience enabling Generative AI solutions in production (eg, RAG-style architectures), including retrieval patterns and evaluation/monitoring practices
• Experience building ingestion frameworks for structured and unstructured data (e.g., event/log, semi-structured JSON), including parsing and enrichment patterns
• Experience designing and scaling ELT/ETL frameworks with orchestration tools such as Airflow (or equivalent)
• Experience implementing data quality, observability, and monitoring practices (e.g., data quality checks, pipeline SLAs/SLOs, alerting)
• Experience with metadata, lineage, and governance concepts and tooling (e.g., data catalogs, lineage, access controls)
• Experience with data modeling best practices for analytics and ML use cases
• Experience with DevOps and CI/CD practices and tools (e.g., GitHub Actions), containerization, and infrastructure-as-code (e.g., Docker, Kubernetes, Terraform)
• Experience supporting ML/AI workflows (feature engineering, data preparation, and model deployment enablement); exposure to MLOps practices is a plus
• Solid SQL skills and experience working with data lakes and warehouses (e.g., Databricks, Snowflake)
• Familiarity with knowledge-centric data approaches (eg, metadata-driven systems, entity resolution, and/or graph concepts) to improve discoverability and downstream analytics
• Solid data quality, observability, and monitoring mindset (profiling, validation, alerting, and reliability improvements)
• Comfort with orchestration, CI/CD, containerization, and infrastructure-as-code (eg, Airflow, GitHub Actions, Docker, Terraform, Kubernetes)
• Demonstrated ability to lead through influence, mentor engineers, and translate ambiguous problems into scalable technical roadmaps
• Demonstrated ability to partner with cross-functional stakeholders, translate requirements into technical solutions, and lead through influence

Preferred Qualifications:

• Solid hands-on engineering in Python and SQL; familiarity with JVM languages (Java/Scala) in Spark ecosystems
• Experience with cloud platforms such as AWS, Azure, or Google Cloud, including managed data services
• Experience with streaming and event-driven architectures (e.g., Kafka, Kinesis, Event Hubs)
• Experience with data quality and validation frameworks (e.g., Great Expectations, Deequ) and/or data observability tooling
• Experience enabling MLOps practices (e.g., feature stores, model registries, experiment tracking, deployment automation)
• Experience with lakehouse architectures, Delta Lake, and advanced Spark optimization/performance tuning
• Experience with data visualization tools and libraries
• Experience with machine learning and predictive analytics
• Familiarity with security and privacy concepts for data platforms (e.g., least privilege, PII/PHI handling) and working with compliance partners

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.