Data Engineering Consultant - Azure Databricks

UnitedHealth Group; Hyderabad; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

UnitedHealth Group in Hyderabad, India seeks a senior Data Engineer to design and operate scalable data pipelines using Databricks, PySpark and Python on Azure. You will build cloud-native solutions with Azure Data Lake and Data Factory, and collaborate with Data Science teams to productionize ML models.

Responsibilities include data quality and governance, on-call production support, Agile participation, CI/CD, and cloud migration from on‑prem to Databricks/Azure.

Qualifications

• 8+ years of SQL/PL-SQL experience.
• 6+ years of hands-on Azure cloud services (ADF, ADLS, Databricks).
• 6+ years of Data Engineering with Python/PySpark.
• 6+ years building ETL/ELT pipelines with Databricks.
• 6+ years in Agile environments.
• 2+ years with Databricks Delta Live Tables (DLT).
• Experience with cloud migration (on-prem → Databricks/Azure).
• Experience with unit testing and pipeline testing frameworks.
• Familiarity with medallion architecture and SCD2 implementations.
• Secure coding practices and vulnerability remediation.
• Exposure to GenAI tools for engineering productivity.
• Ability to analyze logs and troubleshoot production issues.
• Ability to deploy AI-powered solutions responsibly.

Responsibilities

• Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure.
• Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake.
• Collaborate with Data Science and AI teams to operationalize ML models.
• Develop and maintain feature stores, model input pipelines, and real-time streaming frameworks.
• Ensure data quality, governance, and security across the data lifecycle.
• Create reusable frameworks, accelerators, and automation scripts to boost engineering efficiency.
• Optimize performance, scalability, and reliability of data workflows and pipelines.
• Participate in Agile processes, including sprint planning, code reviews, and CI/CD.
• Provide production support and on-call coverage; ensure system stability.
• Comply with company policies, including flexible work arrangements.

Skills

SQL/PL-SQL

Azure

Databricks

Python

PySpark

ETL/ELT

Agile

Delta Live Tables

CI/CD

Security best practices

Education

Bachelor's or Master's degree

Job description

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to startCaring. Connecting. Growing together.

Primary Responsibilities:

• Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure
• Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake
• Collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows
• Develop and maintain feature stores, model input pipelines, and real-time/streaming frameworks
• Ensure data quality, governance, and security across the full data lifecycle
• Build reusable frameworks, accelerators, and automation scripts to improve engineering efficiency
• Optimize performance, scalability, and reliability of data workflows and batch/streaming pipelines
• Participate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelines
• Provide production support and on-call coverage, ensuring system stability and rapid issue resolution
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications:

• Bachelor's or Master's degree
• 8+ years of solid experience in SQL / PL-SQL
• 6+ years of hands‑on experience with Azure cloud services (ADF, ADLS, Databricks)
• 6+ years of experience in Data Engineering with Python/PySpark
• 6+ years of experience in building ETL/ELT pipelines using Databricks
• 6+ years of experience working in Agile environments
• 2+ years of experience with Databricks Delta Live Tables (DLT)
• Experience with cloud migration (on-prem → Databricks/Azure)
• Experience with unit testing, validation, and pipeline testing frameworks
• Familiarity with medallion architecture and SCD2 implementations
• Solid understanding of secure coding practices and vulnerability remediation
• Exposure to GenAI tools (e.g., GitHub Copilot) for engineering productivity
• Proven ability to analyze logs, troubleshoot production issues, and optimize performance
• Demonstrated capability to design and deploy AI-powered solutions responsibly

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.