Data Engineering Consultant - Azure Databricks

Optum; Hyderabad; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Optum Hyderabad seeks a seasoned Data Engineer to design scalable data pipelines with Databricks, Spark, and Python on Azure. You will build cloud-native data lake solutions, integrate ML models into production workflows, and maintain robust feature stores and real-time processing frameworks.

You will ensure data quality, governance, and security while optimizing performance and scalability of batch and streaming pipelines.

Qualifications

• Bachelor's or Master's degree in a relevant field.
• 8+ years of SQL/PL-SQL experience.
• 6+ years with Azure cloud services (ADF, ADLS, Databricks).
• 6+ years Data Engineering with Python/PySpark.
• 6+ years building ETL/ELT pipelines with Databricks.
• 6+ years in Agile environments.
• 2+ years with Databricks Delta Live Tables (DLT).
• Experience with cloud migration (on-prem Databricks/Azure).
• Experience with unit testing and pipeline testing frameworks.
• Familiarity with medallion architecture and SCD2 implementations.
• Secure coding practices and vulnerability remediation.
• Exposure to GenAI tools (e.g., GitHub Copilot).
• Proven logs analysis and production troubleshooting skills.
• Ability to deploy AI-powered solutions responsibly.

Responsibilities

• Design and develop scalable data pipelines using Databricks, Spark, and Python on Azure.
• Build cloud-native solutions leveraging Azure Data Lake, Data Factory, and Delta Lake.
• Collaborate with Data Science and AI teams to operationalize ML models.
• Develop feature stores, model input pipelines, and real-time/streaming frameworks.
• Ensure data quality, governance, and security across the data lifecycle.
• Build reusable frameworks, accelerators, and automation scripts.
• Optimize performance, scalability, and reliability of data workflows.
• Participate in Agile processes: sprint planning, code reviews, CI/CD.
• Provide production support and on-call coverage for stability.
• Adhere to company policies and flexible work arrangements.

Skills

SQL/PL-SQL

Python/PySpark

Azure cloud services

Databricks

Agile

DLT (Delta Live Tables)

Data migration

Unit testing

Security practices

GenAI tools

Log analysis

Education

Bachelor's or Master's degree

Tools

Databricks

Azure Data Factory

Delta Lake

Azure Data Lake Storage

Job description

Primary Responsibilities

• Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure
• Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake
• Collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows
• Develop and maintain feature stores, model input pipelines, and real-time/streaming frameworks
• Ensure data quality, governance, and security across the full data lifecycle
• Build reusable frameworks, accelerators, and automation scripts to improve engineering efficiency
• Optimize performance, scalability, and reliability of data workflows and batch/streaming pipelines
• Participate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelines
• Provide production support and on-call coverage, ensuring system stability and rapid issue resolution
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications

• Bachelor's or Master's degree
• 8+ years of solid experience in SQL / PL-SQL
• 6+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks)
• 6+ years of experience in Data Engineering with Python/PySpark
• 6+ years of experience in building ETL/ELT pipelines using Databricks
• 6+ years of experience working in Agile environments
• 2+ years of experience with Databricks Delta Live Tables (DLT)
• Experience with cloud migration (on-prem Databricks/Azure)
• Experience with unit testing, validation, and pipeline testing frameworks
• Familiarity with medallion architecture and SCD2 implementations
• Solid understanding of secure coding practices and vulnerability remediation
• Exposure to GenAI tools (e.g., GitHub Copilot) for engineering productivity
• Proven ability to analyze logs, troubleshoot production issues, and optimize performance
• Demonstrated capability to design and deploy AI-powered solutions responsibly