Data Engineering Consultant - PySpark, ADF

UnitedHealth Group; Hyderabad; On-site; Confidential; Full time

Job summary

UnitedHealth Group in Hyderabad seeks a Data Engineering Consultant to lead design, development and modernization of data pipelines using PySpark and Databricks, migrating legacy processes to cloud-native frameworks. You will mentor engineers, define architecture standards, and ensure governance across platforms.

You will collaborate with stakeholders to translate business requirements into scalable data solutions, drive CI/CD, and oversee data quality and security in a fast-paced healthcare

Qualifications

• 5+ years of data engineering and ETL/ELT pipeline development experience.
• 4+ years with PySpark, Apache Spark, and Databricks.
• 4+ years of Advanced SQL, data modeling, and performance tuning.
• 3+ years Cloud Data Platform Architecture (ADLS, ADF).
• 3+ years Delta Lake and Lakehouse Architecture experience.
• 3+ years CI/CD, Git and DevOps in data environments.
• Technical leadership including code reviews and design discussions.

Responsibilities

• Lead design, development, and modernization of legacy apps using PySpark and Databricks.
• Define architecture, standards, and best practices for scalable data solutions.
• Mentor data engineers and ensure robust governance, security, and quality.
• Drive migration of legacy processes to cloud-native data frameworks.
• Design and optimize large-scale data pipelines for performance and reliability.
• Collaborate with architects, product owners, and stakeholders to translate requirements.
• Lead CI/CD, automation, and monitoring to improve operations.

Skills

PySpark

Apache Spark

Databricks

SQL

Data modeling

Azure Data Lake Storage

Azure Data Factory

Delta Lake

CI/CD

Git

DevOps

Leadership

ETL/ELT

Python

Agile/Scrum

Healthcare data

Education

Bachelor's degree or equivalent

Tools

Terraform

Kubernetes

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

As a Data Engineering Consultant within our technology organization, you will lead the design, development, and modernization of legacy applications using PySpark and Databricks. You will drive the migration of legacy processes to cloud-native data engineering frameworks and define technical architecture, standards, and best practices for scalable data solutions. In this role, you will collaborate with architects, product owners, and business stakeholders to translate business requirements into technical solutions while mentoring data engineers and ensuring robust data governance, security, and quality across data platforms.

Primary Responsibilities:

• Lead the design, development, and modernization of legacy applications using PySpark and Databricks
• Define technical architecture, standards, and best practices for scalable data solutions
• Mentor and guide data engineers, ensuring high-quality delivery and technical excellence
• Drive migration of legacy processes to cloud-native data engineering frameworks
• Design and optimize large-scale data pipelines for performance, reliability, and scalability
• Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions
• Ensure data governance, security, quality, and compliance across data platforms
• Lead code reviews, technical design discussions, and production support activities
• Implement CI/CD, automation, and monitoring to improve operational efficiency
• Evaluate emerging technologies and drive continuous improvement in data engineering capabilities
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Undergraduate degree or equivalent practical experience
• 5+ years of data engineering and ETL/ELT pipeline development experience
• 4+ years of hands-on experience with PySpark, Apache Spark, and Databricks
• 4+ years of experience with Advanced SQL, data modeling, and performance tuning
• 3+ years of experience with Cloud Data Platform Architecture, including Azure Data Lake Storage (ADLS) and Azure Data Factory (ADF)
• 3+ years of experience working with Delta Lake and Lakehouse Architecture
• 3+ years of experience with CI/CD, Git, and DevOps practices in a data engineering environment
• Technical leadership experience, including leading code reviews and technical design discussions

Preferred Qualifications:

• Experience with Apache Kafka or streaming technologies
• Experience with Azure Synapse Analytics
• Experience with Infrastructure as Code (IaC) using Terraform
• Experience with Machine Learning Pipeline Integration
• Experience with Kubernetes and Containerization
• Experience delivering projects in an Agile/Scrum environment
• Healthcare data domain knowledge
• Familiarity with Data Governance and Data Quality Frameworks
• Solid Python programming skills

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.