Senior Databricks & AI Architect_D3006

Allianz

Pune

On-site

INR 18,000,000 - 26,000,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Allianz is seeking a visionary Senior Databricks & AI Architect to lead a modern data platform and AI roadmap. You will design scalable Spark-based architectures, govern data models, and pioneer Agentic AI workflows to automate claims, underwriting, and risk assessment.

With 8+ years in data engineering/architecture and 3+ years in Databricks leadership, you will drive ROI by optimising clusters, Photon, and cost attribution while ensuring security and governance across regulated insurance data

Qualifications

  • 8+ years in Data Engineering/Architecture with hands-on leadership in the Databricks ecosystem.
  • Expert-level Spark (PySpark/Scala), Delta Lake and Databricks SQL knowledge.
  • Proven AI/ML solution design and deployment; experience with Agentic AI and autonomous agents.
  • FinOps: cost management, instance selection and performance tuning (Z-Ordering, Liquid Clustering).
  • Strong data modelling and reusable ETL/ELT templates for structured/unstructured insurance data.
  • Insurance or FinServ domain familiarity is preferred for policy-centric data understanding.

Responsibilities

  • Architect end-to-end data architectures on the Databricks Data Intelligence Platform.
  • Lead design of AI workflows: Agentic AI, LLM orchestration, and RAG patterns for insurance docs.
  • Own performance and ROI through cluster policies, Photon engine optimisations, and cost-attribution models.
  • Standardise data engineering templates and promote robust data modelling practices.
  • Translate insurance concepts (Premiums, Claims, Actuarial models) into scalable technical schemas and AI logic.
  • Ensure governance and security using Databricks Unity Catalog.

Skills

Databricks Architecture
Leadership in Databricks
Spark (PySpark/Scala)
Delta Lake
Databricks SQL
AI/Agentic AI
FinOps & cost optimisation
Data modelling & ETL/ELT
Insurance domain familiarity

Education

Bachelor's or Master's in CS/Data Science
Databricks Certification (Professional)

Tools

Databricks Unity Catalog
Z-Ordering
Liquid Clustering
LangChain/LlamaIndex

Job description

Position Overview

We are seeking a visionary Senior Databricks & AI Architect to lead the evolution of our modern data platform and AI roadmap. In this role, you will bridge the gap between complex insurance data domains and cutting‑edge AI execution. You will be responsible for designing scalable Spark‑based architectures, implementing rigorous FinOps and optimisation strategies, and pioneering Agentic AI workflows to automate high‑value insurance processes (e.g., claims, underwriting, and risk assessment).

Key Responsibilities
  • Architectural Leadership: Design and govern end‑to‑end data architectures on the Databricks Data Intelligence Platform, ensuring seamless integration with existing insurance ecosystems.
  • AI Innovation: Lead the design of emerging AI workflows, specifically focusing on Agentic AI (multi‑agent systems), LLM orchestration (LangChain/LlamaIndex), and RAG patterns tailored for insurance policy and regulatory documents.
  • Databricks FinOps & Performance: Own the platform’s efficiency. Implement cluster policies, Photon engine optimisations, and cost‑attribution models to ensure high performance with a focus on ROI.
  • Data Engineering Excellence: Standardise data engineering templates (Medallion architecture) and implement robust data modelling (Star/Snowflake, Vault 2.0) to handle structured and unstructured insurance data.
  • Domain Alignment: Translate complex insurance concepts (Premiums, Claims, Actuarial models) into scalable technical schemas and AI logic.
  • Governance & Security: Ensure all designs comply with industry regulations using Databricks Unity Catalog.
Required Skills & Experience
  • Experience: 8+ years in Data Engineering/Architecture, with at least 3+ years of hands‑on leadership within the Databricks ecosystem.
  • Core Tech: Expert‑level knowledge of the Spark framework (PySpark/Scala), Delta Lake, and Databricks SQL.
  • AI/ML Expertise: Proven experience building and deploying AI solutions. Deep understanding of the Agentic AI landscape—moving beyond simple chatbots to autonomous, goal‑oriented agents.
  • FinOps Mastery: Specific experience in Databricks cost management, instance selection, and performance tuning (Z‑Ordering, Liquid Clustering).
  • Data Management: Strong background in data modelling and the creation of reusable ETL/ELT templates and frameworks.
  • Insurance Domain: Prior experience in the Insurance or FinServ sector is highly preferred; you must understand the nuances of policy‑centric data.
Education & Certifications
  • Bachelor’s or Master’s degree in Computer Science, Data Science, or a related field.
  • Preferred: Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.

Allianz Group is one of the most trusted insurance and asset management companies in the world. Caring for our employees, their ambitions, dreams and challenges, is what makes us a unique employer. Together we can build an environment where everyone feels empowered and has the confidence to explore, to grow and to shape a better future for our customers and the world around us.

At Allianz, we stand for unity: we believe that a united world is a more prosperous world, and we are dedicated to consistently advocating for equal opportunities for all. And the foundation for this is our inclusive workplace, where people and performance both matter, and nurtures a culture grounded in integrity, fairness, inclusion and trust.

We therefore welcome applications regardless of ethnicity or cultural background, age, gender, nationality, religion, social class, disability or sexual orientation, or any other characteristics protected under applicable local laws and regulations.

Join us. Let's care for tomorrow.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks & AI Architect_D3006
Senior Databricks & AI Architect_D3006

Allianz Technology • Thiruvananthapuram

On-site
INR 5,000,000 - 9,000,000
Allianz - Senior Databricks & AI Architect
Allianz - Senior Databricks & AI Architect

Allianz PNB Life • Thiruvananthapuram

On-site
INR 2,500,000 - 4,500,000
Assistant Data Architect (Data Scientist)_ 3399
Assistant Data Architect (Data Scientist)_ 3399

Allianz • Pune

On-site
INR 1,800,000 - 2,800,000
Assistant Data Architect (Data Scientist)_ 3399
Assistant Data Architect (Data Scientist)_ 3399

Allianz Technology • Pune District

On-site
INR 1,200,000 - 2,400,000
Allianz - Technical Lead - AI Development
Allianz - Technical Lead - AI Development

Allianz PNB Life • Pune District

On-site
INR 3,500,000 - 7,000,000
Solution Architect [SF & AI]_3451
Solution Architect [SF & AI]_3451

Allianz • Pune

On-site
INR 4,200,000 - 7,000,000
Manager- Business Intelligence
Manager- Business Intelligence

Allianz Services • Pune

On-site
INR 1,200,000 - 2,300,000
Manager- Business Intelligence
Manager- Business Intelligence

Mployee.me • Pune

On-site
INR 1,800,000 - 2,800,000
AI Tech Lead Java &Web_3008
AI Tech Lead Java &Web_3008

Allianz • India

Hybrid
INR 4,000,000 - 6,000,000
Hybrid work model
Pension
Employee shares program
+2
Service Management_3419
Service Management_3419

Allianz Technology • Maharashtra

On-site
INR 1,800,000 - 2,800,000