Data Engineering Consultant - PySpark, ADF

Optum; Hyderabad; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

Optum Hyderabad seeks a Data Engineering Consultant skilled in PySpark and Databricks to lead design, development and modernization of data pipelines. You will migrate legacy processes to cloud-native frameworks, define architecture standards, mentor engineers, and ensure governance, security and quality across data platforms.

This role collaborates with architects, product owners and stakeholders to translate requirements into scalable solutions in a fast-paced environment.

Qualifications

• 5+ years of data engineering and ETL/ELT pipeline development.
• 4+ years PySpark, Apache Spark, and Databricks.
• 4+ years Advanced SQL, data modeling, and performance tuning.
• 3+ years Cloud Data Platform Architecture, including ADLS and ADF.
• 3+ years Delta Lake and Lakehouse Architecture.
• 3+ years CI/CD, Git, and DevOps practices in a data engineering environment.
• Technical leadership experience, including leading code reviews and technical design discussions.

Responsibilities

• Lead design, development, and modernization of legacy apps using PySpark and Databricks.
• Define technical architecture, standards, and best practices for scalable data solutions.
• Mentor data engineers, ensuring high-quality delivery.
• Drive migration of legacy processes to cloud-native data engineering frameworks.
• Design and optimize large-scale data pipelines for performance, reliability, and scalability.
• Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions.
• Ensure data governance, security, quality, and compliance across data platforms.
• Lead code reviews, technical design discussions, and production support activities.
• Implement CI/CD, automation, and monitoring to improve operational efficiency.
• Evaluate emerging technologies and drive continuous improvement in data engineering capabilities.

Skills

PySpark

Databricks

Apache Spark

Advanced SQL

Data modeling

Performance tuning

Azure Data Lake Storage (ADLS)

Azure Data Factory (ADF)

Delta Lake

CI/CD

Git

DevOps practices

Technical leadership

Kafka / streaming

Azure Synapse Analytics

Terraform (IaC)

Kubernetes

Agile/Scrum

Data Governance

Data Quality Frameworks

Education

Undergraduate degree or equivalent practical experience

Tools

Git

CI/CD pipelines

Terraform

Job description

Job Description - Data Engineering Consultant - PySpark, ADF (2387122)

Data Engineering Consultant - PySpark, ADF - 2387122

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

As a Data Engineering Consultant within our technology organization, you will lead the design, development, and modernization of legacy applications using PySpark and Databricks. You will drive the migration of legacy processes to cloud-native data engineering frameworks and define technical architecture, standards, and best practices for scalable data solutions. In this role, you will collaborate with architects, product owners, and business stakeholders to translate business requirements into technical solutions while mentoring data engineers and ensuring robust data governance, security, and quality across data platforms.

Primary Responsibilities:

• Lead the design, development, and modernization of legacy applications using PySpark and Databricks
• Define technical architecture, standards, and best practices for scalable data solutions
• Mentor and guide data engineers, ensuring high-quality delivery and technical excellence
• Drive migration of legacy processes to cloud-native data engineering frameworks
• Design and optimize large-scale data pipelines for performance, reliability, and scalability
• Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions
• Ensure data governance, security, quality, and compliance across data platforms
• Lead code reviews, technical design discussions, and production support activities
• Implement CI/CD, automation, and monitoring to improve operational efficiency
• Evaluate emerging technologies and drive continuous improvement in data engineering capabilities
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re‑assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Undergraduate degree or equivalent practical experience
• 5+ years of data engineering and ETL/ELT pipeline development experience
• 4+ years of hands-on experience with PySpark, Apache Spark, and Databricks
• 4+ years of experience with Advanced SQL, data modeling, and performance tuning
• 3+ years of experience with Cloud Data Platform Architecture, including Azure Data Lake Storage (ADLS) and Azure Data Factory (ADF)
• 3+ years of experience working with Delta Lake and Lakehouse Architecture
• 3+ years of experience with CI/CD, Git, and DevOps practices in a data engineering environment
• Technical leadership experience, including leading code reviews and technical design discussions

Preferred Qualifications:

• Experience with Apache Kafka or streaming technologies
• Experience with Azure Synapse Analytics
• Experience with Infrastructure as Code (IaC) using Terraform
• Experience with Machine Learning Pipeline Integration
• Experience with Kubernetes and Containerization
• Experience delivering projects in an Agile/Scrum environment
• Familiarity with Data Governance and Data Quality Frameworks

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone – of every race, gender, sexuality, age, location and income – deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.

Diversity creates a healthier atmosphere: UnitedHealth Group is an Equal Employment Opportunity/Affirmative Action employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, national origin, protected veteran status, disability status, sexual orientation, gender identity or expression, marital status, genetic information, or any other characteristic protected by law.

UnitedHealth Group is a drug‑free workplace. Candidates are required to pass a drug test before beginning employment.