Databricks AI Architect

Impetus; Chennai District; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Impetus is seeking a hands-on AI Architect with deep Databricks and GenAI expertise to design enterprise-scale data platforms and production AI/ML systems. You will lead architecture decisions, drive implementation across cloud data and AI services, and mentor multiple engineering and data science teams.

The role requires 12–14 years of data/AI experience, strong PySpark/Spark SQL skills, and experience with MLflow, Unity Catalog, and Delta Lake.

Qualifications

• 12–14 years of data/AI/ML experience and large-scale data platform design.
• Deep Databricks architecture with Spark/PySpark and Delta Lake.
• Proven GenAI/LLM solution design including RAG pipelines and embeddings.
• Hands-on Databricks AI capabilities: Mosaic AI, MLflow, and governance via Unity Catalog.
• Strong ML/MLOps: lifecycle, deployment, monitoring, drift/quality evaluation.
• Expert PySpark/Spark SQL and Python pipelines for AI workloads.
• Cloud experience on AWS and Azure; cost optimization and security practices.
• Experience with LangChain or similar multi-agent orchestration.
• Data governance, lineage, security, and responsible AI practices.
• Leadership: setting standards, reviews, and mentorship across teams.
• Excellent stakeholder communication for technical and exec audiences.

Skills

Hands-on AI architecture

Databricks expertise

GenAI / LLM experience

Mosaic AI / Unity Catalog

ML/MLOps foundations

PySpark & Spark SQL

AWS & Azure cloud

LLM orchestration frameworks

Data governance & security

Architecture leadership

Stakeholder communication

Tools

Databricks

Mosaic AI

MLflow

Unity Catalog

Delta Lake

Delta Live Tables

Spark/PySpark

LangChain

Job description

Requirements

The ideal candidate is a hands‑on AI architect with deep Databricks and GenAI expertise, strong data engineering foundations, and a proven record of taking AI solutions to production at enterprise scale. Key requirements:

• 12–14 years of experience in data and AI/ML solutions, including designing and architecting large‑scale (TB/PB) data platforms and production AI/ML systems.


• Deep, hands‑on architecture expertise with Databricks — Spark/PySpark, Delta Lake, lakehouse/medallion architecture, Unity Catalog, Databricks Workflows, and Delta Live Tables — with strong performance and cost optimization.


• Proven experience architecting Generative AI / LLM solutions — RAG pipelines, chunking and embedding strategies, vector store design, prompt engineering, evaluation, and guardrails/responsible‑AI patterns.


• Strong hands‑on experience with native Databricks AI capabilities — Mosaic AI (Model Serving, Vector Search, Agent Framework), Foundation Model APIs, MLflow, Feature Store, and model governance via Unity Catalog.


• Solid ML/MLOps foundations — model lifecycle management, experiment tracking, deployment, monitoring, and drift/quality evaluation in production.


• Expert‑level PySpark and Spark SQL, with strong Python for building reusable data and feature pipelines that power AI workloads (batch and streaming).


• Strong hands‑on experience with AWS and Azure Cloud, including networking, IAM/security, cost optimization, and AI/ML services; comfortable architecting across cloud data and AI services.


• Experience with LLM orchestration frameworks (LangChain/LangGraph or equivalent), agentic patterns, tool use, and multi‑agent workflows at an architecture level.


• Deep expertise in data and AI governance, data quality, lineage, security, and responsible AI — including PII handling, access control (Unity Catalog, RBAC), and audit.


• Proven ability to set architecture standards, lead design reviews, and provide technical leadership and mentorship across multiple engineering and data science teams.


• Strong communication and stakeholder‑management skills; able to drive architecture decisions and translate business needs into technical blueprints for both technical and executive audiences.

Requirements

The ideal candidate is a hands‑on AI architect with deep Databricks and GenAI expertise, strong data engineering foundations, and a proven record of taking AI solutions to production at enterprise scale. Key requirements:

• 12–14 years of experience in data and AI/ML solutions, including designing and architecting large‑scale (TB/PB) data platforms and production AI/ML systems.


• Deep, hands‑on architecture expertise with Databricks — Spark/PySpark, Delta Lake, lakehouse/medallion architecture, Unity Catalog, Databricks Workflows, and Delta Live Tables — with strong performance and cost optimization.


• Proven experience architecting Generative AI / LLM solutions — RAG pipelines, chunking and embedding strategies, vector store design, prompt engineering, evaluation, and guardrails/responsible‑AI patterns.


• Strong hands‑on experience with native Databricks AI capabilities — Mosaic AI (Model Serving, Vector Search, Agent Framework), Foundation Model APIs, MLflow, Feature Store, and model governance via Unity Catalog.


• Solid ML/MLOps foundations — model lifecycle management, experiment tracking, deployment, monitoring, and drift/quality evaluation in production.


• Expert‑level PySpark and Spark SQL, with strong Python for building reusable data and feature pipelines that power AI workloads (batch and streaming).


• Strong hands‑on experience with AWS and Azure Cloud, including networking, IAM/security, cost optimization, and AI/ML services; comfortable architecting across cloud data and AI services.


• Experience with LLM orchestration frameworks (LangChain/LangGraph or equivalent), agentic patterns, tool use, and multi‑agent workflows at an architecture level.


• Deep expertise in data and AI governance, data quality, lineage, security, and responsible AI — including PII handling, access control (Unity Catalog, RBAC), and audit.


• Proven ability to set architecture standards, lead design reviews, and provide technical leadership and mentorship across multiple engineering and data science teams.


• Strong communication and stakeholder‑management skills; able to drive architecture decisions and translate business needs into technical blueprints for both technical and executive audiences.

Good to Have:

Knowledge of LLMOps and observability tooling for AI systems; Databricks / cloud / AI certifications; experience leading cloud/data platform migration and modernization to Databricks; experience contributing to POCs, proposals, and RFPs.

Databricks, AI

AI & GenAI Architecture

• Own the end‑to‑end architecture for AI/ML and GenAI solutions on Databricks — data and feature pipelines, model training/serving, RAG, and agentic applications.


• Design RAG and LLM architectures — retrieval and embedding pipelines, vector store design, prompt/evaluation strategy, and guardrails — using Mosaic AI and Foundation Model APIs.


• Define reference architectures, design patterns, and reusable frameworks for scalable, secure, and cost‑optimized AI platforms.

Data & Platform Foundation

• Architect the lakehouse foundation (Delta Lake, medallion, Unity Catalog) and reusable PySpark pipelines that feed AI and analytics workloads.


• Guide implementation using Databricks Workflows and Delta Live Tables, with standards for data quality, incremental loads, and schema evolution.

MLOps, Governance & Reliability

• Define MLOps/LLMOps practices — model lifecycle, MLflow tracking, deployment, monitoring, and drift/quality evaluation in production.


• Establish AI and data governance, cataloging, lineage, access control, and responsible‑AI standards (PII handling, guardrails, audit) ensuring security and compliance.


• Define observability, monitoring, and cost‑governance practices for reliable and efficient AI systems at scale.

Technical Leadership & Collaboration

• Set and enforce architecture standards and best practices; lead design and code reviews across multiple engineering and data science teams.


• Mentor engineers, ML engineers, and leads, and provide hands‑on guidance from prototype to production readiness.


• Collaborate with product owners, engineering leaders, and business stakeholders to translate requirements into AI blueprints, POCs, and roadmaps.