Data Engineering Consultant - PySpark, ADF

Optum India; Hyderabad; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Optum India in Hyderabad is seeking a senior data engineer to lead the design, development, and modernization of data pipelines using PySpark and Databricks. You will define architecture, mentor engineers, and drive migration to cloud-native frameworks.

The role requires 5+ years in data engineering, strong SQL, and experience with Delta Lake and CI/CD practices. Collaboration with architects and business teams is essential for scalable, governed data solutions.

Qualifications

• Undergraduate degree or equivalent practical experience.
• 5+ years of data engineering and ETL/ELT pipeline development experience.
• 4+ years of hands-on experience with PySpark, Apache Spark, and Databricks.
• 4+ years of experience with Advanced SQL, data modelling, and performance tuning.
• 3+ years of experience with Cloud Data Platform Architecture, including ADLS and ADF.
• 3+ years of experience with Delta Lake and Lakehouse Architecture.
• 3+ years of CI/CD, Git, and DevOps practices in data engineering.

Responsibilities

• Lead design, development and modernization of legacy apps using PySpark and Databricks.
• Define architecture, standards and best practices for scalable data solutions.
• Mentor data engineers to ensure high-quality delivery and technical excellence.
• Drive migration of legacy processes to cloud-native data frameworks.
• Design and optimize large-scale data pipelines for performance and reliability.
• Collaborate with architects and stakeholders to translate requirements into solutions.
• Ensure data governance, security, quality and compliance across platforms.
• Lead code reviews, technical design discussions and production support.
• Implement CI/CD, automation and monitoring to improve operations.
• Evaluate emerging technologies and drive continuous improvement in data engineering.

Skills

PySpark

Apache Spark

Databricks

Advanced SQL

Data Modelling

CI/CD/DevOps

Azure Data Lake Storage

Education

Bachelor's degree in CS or related field

Tools

Terraform

Git

Kubernetes

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

Primary Responsibilities

• Lead the design, development, and modernization of legacy applications using PySpark and Databricks
• Define technical architecture, standards, and best practices for scalable data solutions
• Mentor and guide data engineers, ensuring high-quality delivery and technical excellence
• Drive migration of legacy processes to cloud-native data engineering frameworks
• Design and optimize large-scale data pipelines for performance, reliability, and scalability
• Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions
• Ensure data governance, security, quality, and compliance across data platforms
• Lead code reviews, technical design discussions, and production support activities
• Implement CI/CD, automation, and monitoring to improve operational efficiency
• Evaluate emerging technologies and drive continuous improvement in data engineering capabilities
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications

• Undergraduate degree or equivalent practical experience
• 5+ years of data engineering and ETL/ELT pipeline development experience
• 4+ years of hands-on experience with PySpark, Apache Spark, and Databricks
• 4+ years of experience with Advanced SQL, data modelling, and performance tuning
• 3+ years of experience with Cloud Data Platform Architecture, including Azure Data Lake Storage (ADLS) and Azure Data Factory (ADF)
• 3+ years of experience working with Delta Lake and Lakehouse Architecture
• 3+ years of experience with CI/CD, Git, and DevOps practices in a data engineering environment
• Technical leadership experience, including leading code reviews and technical design discussions

Preferred Qualifications

• Experience with Apache Kafka or streaming technologies
• Experience with Azure Synapse Analytics
• Experience with Infrastructure as Code (IaC) using Terraform
• Experience with Machine Learning Pipeline Integration
• Experience with Kubernetes and Containerization
• Experience delivering projects in an Agile/Scrum environment
• Healthcare data domain knowledge
• Familiarity with Data Governance and Data Quality Frameworks
• Solid Python programming skills

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.