Data Engineer (Databricks + AI)

Durapid Technologies Pvt Ltd; Bengaluru Urban; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Durapid Technologies Pvt Ltd in Bengaluru seeks an experienced AI/Data Engineer to design scalable data pipelines and AI-powered apps. You will work with Databricks, PySpark, and cloud platforms to deliver production-grade AI solutions and data services.

Responsibilities include building REST APIs with FastAPI/Flask, implementing RAG with enterprise knowledge bases, and collaborating with data scientists and stakeholders to optimize inference pipelines.

Qualifications

• 3+ years of data engineering with cloud Databricks and PySpark/Python.
• Experience building ETL/ELT pipelines at scale.
• Hands-on with LangChain and LlamaIndex in LLM workloads.
• Knowledge of RAG architectures and enterprise KB integration.
• Strong Python API development (FastAPI/Flask) and CI/CD.

Responsibilities

• Design, develop, and optimize scalable data pipelines on Azure/AWS/GCP.
• Develop AI-powered applications using LLMs, LangChain, and LlamaIndex.
• Implement RAG solutions with enterprise knowledge bases.
• Create REST APIs for AI and data services.
• Collaborate with Data Scientists, ML Engineers, and Business stakeholders.
• Optimize model performance, prompt engineering, and inference pipelines.
• Follow DevOps and MLOps practices for deployment, monitoring, and maintenance.

Skills

Databricks

PySpark

Python

AWS

Azure

GCP

ETL/ELT pipelines

LLM frameworks

LangChain

LlamaIndex

RAG

FastAPI/Flask

API development

MLOps

DevOps

Communication

Tools

LangChain

LlamaIndex

FastAPI

Flask

Job description

Job Summary

We are looking for an experienced AI / Data Engineer to design and build scalable data pipelines and AI-powered applications. The ideal candidate will combine strong data engineering fundamentals with hands-on experience in LLM frameworks, Retrieval-Augmented Generation (RAG), and API development to deliver production-grade AI and data solutions.

Key Responsibilities :

• Design, develop, and optimize scalable data pipelines using Azure/AWS/GCP Databricks and PySpark.
• Develop AI-powered applications using LLMs, LangChain, and LlamaIndex.
• Design and implement RAG (Retrieval-Augmented Generation) solutions using enterprise knowledge bases.
• Develop REST APIs using Python (FastAPI/Flask) for AI and data services.
• Collaborate with Data Scientists, ML Engineers, and Business stakeholders to deliver AI-driven solutions.
• Optimize model performance, prompt engineering, and inference pipelines.
• Follow DevOps and MLOps best practices for deployment, monitoring, and maintenance.

Required Skills & Qualifications :

• 3+ years of experience in data engineering, with strong expertise in AWS/Azure/GCP Databricks and PySpark/Python.
• Proven experience building and maintaining ETL/ELT pipelines at scale.
• Hands-on experience with LLM frameworks such as LangChain and LlamaIndex.
• Practical knowledge of RAG architecture and enterprise knowledge base integration.
• Strong Python skills, including API development using FastAPI or Flask.
• Familiarity with prompt engineering and inference pipeline optimization.
• Understanding of DevOps/MLOps practices (CI/CD, model deployment, monitoring).
• Strong communication and collaboration skills to work across cross-functional teams.