Data Engineering Lead-Azure Data bricks, SQl,Python,Spark,Airflow

Optum India; Bengaluru; In loco; INR 1.500.000 - 2.100.000; Tempo pieno

Descrizione del lavoro

Optum India is seeking a skilled Data Engineer to design, develop, and operate scalable data pipelines on Azure Databricks, Spark SQL, Python, and Airflow. You will build ingestion, transformation, and quality checks across diverse data sources, optimizing performance and enabling analytics, AI, and reporting.

Ideal candidates have 4+ years in modern data engineering, experience with Azure services (ADF, Key Vault), and a strong foundation in SQL, Spark, Delta Lake, and ETL/ELT practices.

Competenze

• Bachelor's degree in engineering or equivalent.

Mansioni

• Design, develop, test, and deploy scalable data pipelines using Azure Databricks, Spark SQL, Python, and Airflow
• Build and maintain data ingestion, transformation, and data curation processes across multiple enterprise data sources
• Develop complex SQL queries and optimize data processing workflows for performance and scalability
• Create, maintain, and support Airflow DAGs for orchestration, scheduling, monitoring, and error handling
• Collaborate with Product Owners, Business Analysts, and stakeholders during requirements gathering, backlog refinement, and sprint planning activities
• Implement data quality validations, reconciliation checks, and exception handling mechanisms
• Support the development and maintenance of Delta Lake and Lakehouse architectures within Databricks
• Participate in data platform modernization and cloud migration initiatives
• Develop reusable frameworks, utilities, and automation solutions to improve engineering efficiency
• Troubleshoot pipeline failures, performance bottlenecks, and complex data issues
• Follow coding standards, security requirements, and engineering best practices
• Contribute to CI/CD implementation and automated deployment processes
• Work closely with cross-functional teams to support analytics, reporting, AI, and business operational needs
• Prepare and maintain technical documentation, design specifications, and operational runbooks
• Support production deployments, incident resolution, root cause analysis, and operational excellence initiatives
• Utilize AI-assisted development tools and automation capabilities to improve delivery efficiency and code quality
• Provide technical recommendations and participate in design reviews and architecture discussions
• Demonstrate ownership, accountability, and commitment to project deliverables
• Stay current with emerging technologies, cloud services, and modern data engineering practices
• Comply with company policies, security standards, and organizational guidelines
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Conoscenze

Azure Databricks

Apache Spark

Spark SQL

SQL Development

Python Programming

Apache Airflow

Delta Lake

Data Quality

Metadata & Lineage

Data Ingestion

Data Transformation

ETL/ELT

Data Warehousing

Data Modeling

Azure Data Factory

Azure Data Lake Storage

CI/CD Pipelines

GitHub/GitHub Actions

Formazione

Bachelor's Degree in Engineering

MCA or equivalent

Computer Science/Information Technology

Strumenti

Azure Data Factory

Azure Key Vault

Azure DevOps or GitHub

GitHub Actions

Snowflake

Descrizione del lavoro

• Data quality implementation

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale.

Join us to start Caring. Connecting. Growing together.

Primary Responsibilities

• Design, develop, test, and deploy scalable data pipelines using Azure Databricks, Spark SQL, Python, and Airflow
• Build and maintain data ingestion, transformation, and data curation processes across multiple enterprise data sources
• Develop complex SQL queries and optimize data processing workflows for performance and scalability
• Create, maintain, and support Airflow DAGs for orchestration, scheduling, monitoring, and error handling
• Collaborate with Product Owners, Business Analysts, and stakeholders during requirements gathering, backlog refinement, and sprint planning activities
• Implement data quality validations, reconciliation checks, and exception handling mechanisms
• Support the development and maintenance of Delta Lake and Lakehouse architectures within Databricks
• Participate in data platform modernization and cloud migration initiatives
• Develop reusable frameworks, utilities, and automation solutions to improve engineering efficiency
• Troubleshoot pipeline failures, performance bottlenecks, and complex data issues
• Follow coding standards, security requirements, and engineering best practices
• Contribute to CI/CD implementation and automated deployment processes
• Work closely with cross-functional teams to support analytics, reporting, AI, and business operational needs
• Prepare and maintain technical documentation, design specifications, and operational runbooks
• Support production deployments, incident resolution, root cause analysis, and operational excellence initiatives
• Utilize AI-assisted development tools and automation capabilities to improve delivery efficiency and code quality
• Provide technical recommendations and participate in design reviews and architecture discussions
• Demonstrate ownership, accountability, and commitment to project deliverables
• Stay current with emerging technologies, cloud services, and modern data engineering practices
• Comply with company policies, security standards, and organizational guidelines
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications

• Bachelor's Degree in Engineering, Computer Science, Information Technology, MCA, or equivalent experience
• 4+ years of relevant hands-on experience in Azure Databricks and Modern Data Engineering
• Experience working in Agile/Scrum delivery environments
• Technical Skills:
  • Data Engineering Expertise
    • Data ingestion frameworks
    • Data transformation and enrichment
    • Batch processing pipelines
    • Data quality implementation
    • Query optimization
    • Performance tuning
    • Metadata and lineage concepts
  • Cloud Technologies knowledge
    • Microsoft Azure
    • Azure Data Factory (ADF)
    • Azure Key Vault
    • Azure DevOps or GitHub
    • CI/CD Pipelines
  • Proven core skills
    • Azure Databricks
    • Apache Spark
    • Spark SQL
    • SQL Development
    • Python Programming
    • Apache Airflow
    • DAG Development and Workflow Orchestration
    • Delta Lake
    • Azure Data Lake Storage (ADLS)
    • ETL / ELT Development
    • Data Warehousing Concepts
    • Data Modeling Fundamentals

Preferred Qualifications

• Azure Databricks certification or related Azure certifications
• Experience working with Healthcare, Claims, Member, Provider, Payment Integrity, or Affordability data domains
• Experience with Snowflake and modern cloud data platforms
• Experience using GitHub Actions, DevOps automation, and Infrastructure as Code
• Healthcare payer industry experience
• Experience supporting enterprise data platforms and analytics ecosystems
• Experience with Payment Integrity, Risk Adjustment, Claims Analytics, or Provider Analytics
• Exposure to AI/ML data engineering workflows and AI-enabled analytics solutions
• Exposure to Generative AI, Azure OpenAI, or AI-assisted engineering tools
• Understanding of Data Lakehouse, Data Mesh, or Data Fabric concepts
• Exposure to AI-driven development and engineering productivity tools

Key Competencies

• Solid analytical and problem-solving skills
• Ability to independently deliver medium to complex technical solutions
• Solid debugging and troubleshooting capabilities
• Excellent written and verbal communication skills
• Effective stakeholder collaboration and teamwork
• Solid learning agility and growth mindset
• Ability to work in a fast-paced and dynamic environment
• Self-motivated with solid ownership and accountability

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.