Associate Data Engineering - Databricks

Optum; Bengaluru; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Optum in Bengaluru, India, seeks a data & AI engineer to design, develop, and maintain scalable data pipelines using Databricks and PySpark. You will orchestrate batch, streaming, and incremental processing with Airflow and build ELT pipelines in Snowflake for reliable performance and cost efficiency.

Responsibilities include developing production-grade Python code, building APIs and AI services, and deploying ML models with Azure-native tools.

Qualifications

• Graduate degree or equivalent experience.
• Hands-on experience with Databricks (Spark / PySpark) in production.
• Experience building and hosting REST APIs and AI services in Python.
• Experience with Azure AI Services, Azure OpenAI or Azure ML.
• Experience with LLMs, Generative AI, RAG, vector databases and embeddings.
• Experience with AI governance, privacy and security practices.
• Experience with CI/CD pipelines, Git and IaC deployments.
• Snowflake data modeling, optimization, and performance tuning.
• Azure cloud architecture and cloud-native patterns.
• Knowledge of Agentic AI frameworks and intelligent workflow orchestration.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / PySpark).
• Orchestrate workflows with Apache Airflow for batch/streaming processing.
• Build and optimize ELT pipelines in Snowflake for performance and cost efficiency.
• Develop production-quality Python code for data processing and AI workloads.
• Design and host APIs, AI services, and data products using Python frameworks.
• Deploy APIs, ML models, and AI services using Azure-native services.
• Implement AI/ML and Generative AI solutions including MLOps practices.
• Collaborate with cross-functional teams to build AI-powered data solutions.
• Ensure compliance with company policies and security practices.

Skills

Databricks / PySpark

REST APIs

Python

Azure AI Services

LLMs / Generative AI

CI/CD / IaC

Snowflake

Azure Cloud

Agentic AI frameworks

Education

Graduate degree or equivalent experience

Tools

Snowflake

Azure

LangChain

Semantic Kernel

Job description

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / PySpark)
• Orchestrate and monitor workflows using Apache Airflow for batch, streaming, and incremental processing
• Build and optimize ELT pipelines in Snowflake, ensuring performance, reliability, and cost efficiency
• Develop production-quality Python code for data processing, transformations, integrations, and AI/ML workloads
• Design, build, and host APIs, AI services, and data products using modern Python frameworks
• Deploy and manage APIs, ML models, and AI services using Azure-native services (Azure Functions, App Services, API Management, Azure AI Services)
• Design and implement AI/ML and Generative AI solutions, including model integration, prompt engineering, RAG (Retrieval-Augmented Generation), and intelligent workflow automation
• Build and manage vector databases, embeddings pipelines, semantic search capabilities, and knowledge retrieval frameworks
• Develop and operationalize ML pipelines, feature engineering workflows, model monitoring, and MLOps practices
• Leverage LLMs and Agentic AI frameworks to automate data onboarding, data quality validation, metadata management, and business processes
• Apply Azure cloud best practices for security, scalability, monitoring, resiliency, governance, and cost optimization
• Implement data quality checks, lineage tracking, observability, model evaluation, and AI governance controls
• Collaborate with Data Science, Analytics, Product, and Platform teams to build AI-powered products and intelligent data solutions
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications

• Graduate degree or equivalent experience
• Solid hands-on experience with Databricks (Spark / PySpark) in production environments
• Experience building and hosting REST APIs, AI services, and microservices using Python frameworks
• Experience with Azure AI Services, Azure OpenAI, Azure Machine Learning, or equivalent AI platforms
• Experience working with Large Language Models (LLMs), Generative AI, RAG architectures, vector databases, embeddings, and prompt engineering
• Experience implementing AI governance, model evaluation, privacy, security, and responsible AI practices
• Experience with CI/CD pipelines, Git-based version control, Infrastructure as Code, and deployment automation
• Solid working experience with Snowflake, including data modeling, optimization, and performance tuning
• Solid experience with Microsoft Azure, including cloud-native architecture and best practices
• Knowledge of Agentic AI frameworks (e.g., Semantic Kernel, LangChain, AutoGen, CrewAI, or similar) and intelligent workflow orchestration