Data Engineering Lead

Optum

Bengaluru

On-site

INR 1,400,000 - 2,400,000

Full time

10 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance

Job summary

Optum in Bengaluru, India seeks a seasoned Data Engineer to design and implement data-intensive applications, leveraging Gen AI LLMs for agentic data engineering projects. You will work with solution architects to evaluate requirements, develop ETL pipelines, and build scalable data lakes/warehouses using Azure Databricks, Synapse, and Snowflake.

You will champion DevOps practices, automation, and resilient architectures, ensuring secure, high-quality data platforms while collaborating across

Qualifications

  • B.Tech in computer science or similar field with minimum 5 years' experience of Data Engineering and at least 1 year' experience leveraging Gen AI LLM models to design agentic solutions in data engineering space.
  • Experience in ETL/Pipeline Development using Azure Databricks/Apache Spark and Azure Data Factory with development expertise on batch and real-time data integration.
  • Experience in programming using Python.
  • Experience in data ingestion, preparation, integration and operationalization techniques in addressing data requirements.
  • Experience in Cloud data warehouse like Azure Synapse, Snowflake analytical warehouse.
  • Experience with Orchestration tools, Azure DevOps and GitHub.
  • Experience in building end to end architecture for Data Lakes, Data Warehouses and Data Marts.
  • Experience in relational data processing tech like MS SQL, Delta Lake, Spark SQL.
  • Own end-to-end development including coding, testing, debugging and deployment.
  • Knowledge of Reliability Engineering, Deployment planning, Fault Tolerant architecture, Test Automation.
  • Knowledge of ETL and Data Warehousing concepts, strategies, methodologies.
  • Ability to provide forward-thinking data and analytics solutions.
  • Strong teamwork, prioritization, adaptability, and communication skills.
  • Familiarity with Azure services like Azure functions, Azure Data Lake Store, Azure Cosmos.
  • Familiarity with Healthcare business data models.

Responsibilities

  • Work with solution architects and software engineers to evaluate requirements and deliver high-quality solutions.
  • Design, develop, test and maintain data-intensive applications including ETL, data storage, and data app development.
  • Leverage Gen AI LLM models to design agentic data engineering solutions.
  • Follow and improve continuous delivery and DevOps processes within the team.
  • Build and maintain configuration management, deployment strategies, and monitoring tools.
  • Collaborate on quality strategies to ensure data platform correctness, resilience, scalability and security.
  • Participate in system analysis, design walkthroughs, and code reviews.
  • Participate in defect review and triage.
  • Adhere to design and coding standards and improve delivery methods.
  • Perform technical analysis of existing reporting solutions for performance and quality improvements.
  • Drive continuous improvement and automation in data engineering processes.
  • Maintain production products, solutions, documentation, and environments; resolve production issues promptly.
  • Comply with company policies and changes including work location, team assignments, work schedules, and flexible work arrangements.

Skills

Python
ETL/Pipeline development
Gen AI / LLM apps
Azure Databricks
Azure Data Factory
Azure Synapse
Snowflake
Delta Lake
MS SQL
Spark SQL
DevOps
Data Lakes/Warehouses

Education

B.Tech in Computer Science or similar

Tools

Azure Databricks
Azure Data Factory
Azure Synapse
Snowflake
Delta Lake
MS SQL
GitHub
Azure DevOps

Job description

Primary Responsibilities
  • Work with solution architects and other software engineers to evaluate functional and non-functional requirements and deliver creative, high-quality solutions
  • Design, develop, test and maintain data intensive applications including ETL, relational and non-relational data storage, data application development
  • Leverage AIDLC & design Agentic data engineering solution by using available Gen AI & LLM models
  • Follow and improve on processes for continuous delivery and DevOps within your team
  • Build and main configuration management and automation tools, deployment strategies/processes, monitoring tools
  • Collaborate on quality strategies that ensure our data platform is correct, resilient, scalable and secure
  • Participate in and provide input for system analysis, design walkthroughs and code reviews
  • Participating in defect review and triage
  • Adhere to design/coding standards and constantly improve the way we build and deliver software
  • Technical analysis of existing reporting solutions to identify opportunities for improvement both from a best practice and quality/ performance improvement perspective
  • Drive continuous improvement and automation solutions and processes
  • Maintain production products, solutions, documentation, and environments to ensure optimal health, including resolving production issues and responding quickly to priority problems
  • Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
Required Qualifications
  • B.Tech required in computer science or similar field with minimum 5 years' experience of Data Engineering and at least 1 years' experience leveraging Gen AI LLM models to design agentic solutions in data engineering space
  • Experience in ETL/Pipeline Development using tools such as Azure Databricks/Apache Spark and Azure Data Factory with development expertise on batch and real-time data integration
  • Experience in programming using Python
  • Experience in data ingestion, preparation, integration and operationalization techniques in optimally addressing the data requirements
  • Experience in Cloud data warehouse like Azure Synapse, Snowflake analytical warehouse
  • Experience with Orchestration tools, Azure DevOps and GitHub
  • Experience in building end to end architecture for Data Lakes, Data Warehouses and Data Marts
  • Experience in relational data processing technology like MS SQL, Delta Lake, Spark SQL
  • Experience to own end-to-end development, including coding, testing, debugging and deployment
  • Experienced in Software Engineering practices like Reliability Engineering, Deployment planning, Fault Tolerant architecture, Test Automation
  • Extensive knowledge of ETL and Data Warehousing concepts, strategies, methodologies
  • Ability to provide solutions that are forward-thinking in data and analytics
  • Proven team oriented with solid collaboration, prioritization, and adaptability skills
  • Proven excellent written and verbal communication skills including presentation skills
  • Familiarity with Azure services like Azure functions, Azure Data Lake Store, Azure Cosmos
  • Familiarity with Healthcare business data models
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer - Senior 1
Data Engineer - Senior 1

Fa Espx Saasfaprod1 • Pune District

On-site
INR 1,200,000 - 1,800,000
Data Engineer 10
Data Engineer 10

Fa Espx Saasfaprod1 • Pune District

On-site
INR 1,500,000 - 2,300,000
Technical Lead - Data Engineer (Data&AI)
Technical Lead - Data Engineer (Data&AI)

Srijan: Now Material • Gurugram District

On-site
INR 4,000,000 - 7,500,000
Lead Engineer - Data & AI
Lead Engineer - Data & AI

Quest Global • Bengaluru

On-site
INR 1,000,000 - 1,500,000
Azure Data Engineer Lead/Architect
Azure Data Engineer Lead/Architect

Creative Synergies Group • Bengaluru

On-site
INR 1,500,000 - 2,500,000
Data Engineer Staff
Data Engineer Staff

Hartford • Hyderabad

Hybrid
INR 1,200,000 - 2,000,000
Associate Director
Associate Director

Axtria - Ingenious Insights • Dadri

On-site
INR 900,000 - 1,500,000
Data Engineer
Data Engineer

Ashley Global Capability Center • Chennai

On-site
INR 7,005,253 - 10,507,880
Azure Data Engineer
Azure Data Engineer

NetConnectGlobal • Bengaluru

Hybrid
INR 1,800,000 - 2,800,000
Agentic Data Delivery Lead
Agentic Data Delivery Lead

ExlService Holdings, Inc. • Pune District

On-site
INR 9,000,000 - 15,000,000