Data Engineer - Databricks & Agentic AI

EXL

Gurugram District

On-site

INR 1,200,000 - 2,400,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Cloud platform exposure
Career development path
Cross-industry projects

Job summary

EXL is seeking a hands-on Data Engineer with strong Databricks experience to design and build lakehouse architectures, data pipelines, data products, and governed data platforms across AWS, Azure, and GCP.

The role emphasizes production-grade implementation, data governance, and agentic AI capabilities, with a foundation in Python, SQL, Spark, and dbt. Based in India with exposure to global cloud environments.

Qualifications

  • Proficiency in Python and SQL for data pipelines and transformations.
  • Solid Spark/PySpark experience with production-grade data processing.
  • Experience with lakehouse architectures and cloud object storage.
  • Familiarity with data governance, lineage, and modeling best practices.

Responsibilities

  • Design, build, and maintain scalable lakehouse architectures and data products using Databricks, Delta Lake, Apache Spark, Python, SQL, and cloud object storage.
  • Build reliable ingestion pipelines for APIs, files, SFTP, databases, and third‑party data sources using Databricks‑native and Python tooling.
  • Develop reusable transformation frameworks and governed data models with data quality controls, orchestration, and monitoring across Bronze, Silver, and Gold layers.
  • Configure and optimize within Databricks environments, including workspaces, clusters, jobs, notebooks, and deployment workflows.
  • Implement data governance, discoverability, lineage, access controls, and secure sharing using Unity Catalog.
  • Build data‑aware agents and multi‑agent workflows that reason over governed data and invoke approved models and tools.
  • Operate data pipelines and agent workloads with Git, CI/CD, automated testing, and production monitoring.

Skills

Python
SQL
Spark
Data modelling
Databricks

Tools

Delta Lake
dbt
Unity Catalog

Job description

We are looking for a hands‑on Data Engineer with strong Databricks experience to join our product and solution engineering team. The primary focus of this role is to design and build reliable lakehouse architectures, data pipelines, data products, and governed data platforms on Databricks across AWS, Azure, and GCP environments.

In addition to core data engineering, the role offers the opportunity to build agentic AI capabilities within Databricks. This includes developing data‑aware agents and multi‑agent workflows that use governed enterprise data, models, functions, and tools through Unity Catalog and Databricks AI governance capabilities.

This is a Data Engineering role, not a generalist Applied AI role. We are looking for someone whose foundation is Python, SQL, Spark, data modelling, pipeline engineering, and Databricks platform implementation, with the technical depth and interest to extend those skills into production‑grade agentic systems.

Key Responsibilities
  • Design, build, and maintain scalable lakehouse architectures and data products using Databricks, Delta Lake, Apache Spark, Python, SQL, and cloud object storage.
  • Build reliable ingestion pipelines for APIs, files, SFTP, databases, and third‑party data sources using Databricks‑native and Python‑based tooling.
  • Develop reusable transformation frameworks and governed data models using Databricks SQL, PySpark, and dbt, with data quality controls, orchestration, and monitoring across Bronze, Silver, and Gold layers.
  • Configure and work effectively within Databricks environments, including workspaces, compute, clusters or serverless compute, jobs, notebooks, repositories, secrets, service principals, and deployment workflows.
  • Implement data governance, discoverability, lineage, access controls, and secure sharing using Unity Catalog.
  • Build data‑aware agents and multi‑agent workflows within Databricks that can reason over governed enterprise data and invoke approved models, functions, APIs, and tools.
  • Use Databricks model serving and AI gateway capabilities to manage model access, routing, usage controls, observability, and guardrails for agentic applications.
  • Operationalise data pipelines and agentic workloads through Git, CI/CD, automated testing, environment promotion, and production monitoring.
  • Work with solution architects, application engineers, data scientists, and client teams to move solutions from prototypes into secure and supportable production deployments.
  • Optimise Databricks workloads for performance, reliability, and cloud cost efficiency.
What We’re Looking For

You do not need to be an expert in every AI framework. Strong Data Engineering and Databricks fundamentals are the priority; agentic AI experience is an important differentiator.

  • Approximately 3-6 years of relevant experience in Data Engineering, Data Platform Engineering, or a closely related role.
  • Strong hands‑on experience with Databricks in a real development or production environment; familiarity with Databricks setup and administration is essential.
  • Strong programming skills in Python, SQL, and PySpark, including performance‑aware Spark development and debugging.
  • Experience designing and implementing lakehouse architectures using Delta Lake and medallion or equivalent layered patterns.
  • Hands‑on experience with dbt for modular SQL transformations, testing, documentation, lineage, and deployment within a modern data platform.
  • Experience building production‑grade data ingestion, transformation, orchestration, validation, and monitoring pipelines.
  • Practical experience with Unity Catalog, including catalogs, schemas, tables, volumes, permissions, lineage, and governance patterns.
  • Understanding of Databricks Workflows or Jobs, cluster and compute configuration, secrets, service principals, repositories, and environment promotion.
  • Experience with Git and CI/CD practices for data engineering workloads, including automated testing and deployment.
  • Experience with at least one cloud platform: AWS, Azure, or GCP, including object storage, identity, networking, and managed data services.
  • Strong troubleshooting skills and the ability to work across data, platform, security, and application boundaries.
Agentic AI Experience
  • Hands‑on experience building LLM‑powered applications, agents, or tool‑calling workflows using Python and a suitable framework or SDK.
  • Ability to design multi‑agent systems with clear orchestration, state, tool permissions, failure handling, human approval points, and observability.
  • Experience grounding agents on governed enterprise data through SQL, vector search, retrieval, functions, APIs, or other controlled tools.
  • Understanding of model serving, prompt and response logging, evaluation, tracing, token and cost controls, rate limits, and safety guardrails.
  • Familiarity with registering and governing models, agents, functions, connections, and tools through Unity Catalog and Databricks AI governance capabilities.
  • Ability to distinguish a production agent from a demo by addressing identity, permissions, data access, testing, monitoring, resilience, and operational support.
Nice to Have
  • Experience with Databricks Asset Bundles, Databricks Connect, MLflow, Model Serving, Vector Search, Lakehouse Monitoring, or Delta Live Tables / Lakeflow Declarative Pipelines.
  • Experience with Mosaic AI Agent Framework, LangGraph, LangChain, Semantic Kernel, or another orchestration framework used for production agent systems.
  • Experience integrating dbt with Databricks workflows, Unity Catalog, Git‑based development, and CI/CD; exposure to Snowflake, Microsoft Fabric, or Airflow is also useful.
  • Experience building SaaS or multi‑tenant applications, data APIs, or governed client‑facing data products.
  • Exposure to Docker, serverless containers, cloud networking, private connectivity, firewalls, and enterprise security requirements.
What Success Looks Like
  • You can independently build and troubleshoot a Databricks data pipeline from ingestion through curated, governed consumption layers.
  • You understand how a Databricks workspace should be organised and secured, not only how to write code in a notebook.
  • You can build an agent that uses governed data and tools, and you can explain how its identity, permissions, model access, logging, evaluation, and production support will work.
  • You create reusable engineering patterns that help multiple product teams deploy data and agentic solutions faster and more safely.
What We Offer
  • Opportunity to work on data, analytics, AI, and digital products across Retail & CPG, Media & Entertainment, Sports, Energy, Telecommunications, and other industries.
  • Hands‑on work across the Databricks platform, from lakehouse engineering and governance through production‑grade agentic AI solutions.
  • Exposure to AWS, Azure, GCP, Databricks, Snowflake, and modern cloud‑native technologies.
  • Opportunity to help establish reusable Databricks and agentic engineering practices for a growing product and solution organisation.
  • A development path toward Senior Data Engineer, Databricks Platform Engineer, Data Architect, or technical leadership roles.

About EXL

EXL (NASDAQ:EXLS) is a global data and AI company that helps clients improve business outcomes through data, analytics, AI, digital solutions, and operations. Our teams work with organisations across industries to design, build, and deploy technology‑driven solutions addressing complex business problems. Headquartered in New York, EXL has more than 40,000 professionals in locations throughout the United States, Europe, Asia (primarily India and Philippines), Latin America, Australia and South Africa.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

1984760-Assistant Vice President
1984760-Assistant Vice President

ExlService Holdings, Inc. • Gurugram District

On-site
INR 4,000,000 - 6,000,000
Applied AI Engineer - Senior Consultant
Applied AI Engineer - Senior Consultant

EXL • Pune District, Gurugram District, Bengaluru

On-site
INR 420,000 - 900,000
1984760-Assistant Vice President
1984760-Assistant Vice President

EXL • Gurugram District

On-site
INR 3,500,000 - 6,500,000
Applied AI Engineer - Agentic Systems
Applied AI Engineer - Agentic Systems

EXL • Gurugram District

On-site
INR 2,000,000 - 3,200,000
Senior Databricks Engineer
Senior Databricks Engineer

ExlService Holdings, Inc. • Pune District

On-site
INR 1,500,000 - 1,900,000
AWS Data Engineer
AWS Data Engineer

EXL • Bengaluru

Hybrid
INR 2,500,000 - 4,000,000
2702 - Data Engineer
2702 - Data Engineer

EXL • Pune City

On-site
INR 1,000,000 - 1,500,000
Data Engineer - Databricks
Data Engineer - Databricks

Digit88 Technologies • Bengaluru

On-site
INR 1,200,000 - 1,800,000
Flexible work model
Opportunity to work on cutting-edge AI products
Technical Data Engineer -Databricks
Technical Data Engineer -Databricks

Simfluent • Coimbatore

On-site
INR 1,200,000 - 1,800,000
Competitive salary
Insurance package
Learning & development resources
Technical Data Engineer -Databricks
Technical Data Engineer -Databricks

ShyftLabs • Coimbatore District

On-site
INR 1,200,000 - 2,500,000
Competitive salary
Insurance package
Learning & development resources