Lead Databricks Solution Architect [T500-29584]

Talent500; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Zurich Insurance’s Core Insurance Platforms (CIP) seeks a Fin Enhance Databricks Engineer to design and scale a Databricks-based platform for global financial data integration, combining hands-on engineering with SAP finance integration.

Two profiles exist: senior/architecture or hands-on engineering and delivery, both leveraging AI daily, with emphasis on governance, data quality, and production readiness.

Qualifications

• Strong hands-on experience with Databricks platform.
• Expertise in Unity Catalog, Workflows, Delta Lake and Databricks assets.
• Experience across DEV/UAT/PROD Databricks workspaces.
• Proficiency in Python, PySpark, Spark SQL.

Responsibilities

• Design, build and maintain Databricks-based pipelines for financial data ingestion and posting preparation.
• Contribute to architecture standards covering Delta Lake, Unity Catalog and CI/CD workflows.
• Migrate legacy integrations to Databricks/PySpark solutions and ensure reliability.

Skills

Databricks

Python

PySpark

SQL

CI/CD

Git

Dash/Plotly

Tools

Delta Lake

Unity Catalog

Medallion Architecture

Auto Loader

Spark SQL

Job description

Who are we:

Core Insurance Platforms (CIP) is Zurich’s global capability responsible for building, running, and evolving core insurance technology. We set a unified, scalable operating model—covering governance, standards, architecture, service delivery, and reuse—so our business units can deliver at speed and scale.

CIP is the strategic steward of Zurich’s Guidewire ecosystem, aligning platform roadmaps to business strategy while driving stability, modernization, reduced supplier dependency, and long term cost efficiency.

India delivery center is one of our global delivery and capability hub. We bring together experts in AI, engineering, analysis, quality, and architecture to deliver product & process solutions, application run services, change and transformation initiatives, and centralized platform services across both on prem and Guidewire Cloud environments. Our teams operate from multiple global delivery centers, supporting Zurich’s business units worldwide.

What is the Financial Control - SAP Integration Team

The Financial Control - SAP Integration Team provides end-to-end capabilities for the reliable exchange of financial data between core finance platforms, SAP S/4HANA and enterprise source systems. The team designs, delivers and operates finance-critical integration solutions with strong focus on data quality, posting accuracy, governance and operational resilience.

The team is leading the transition from legacy technologies towards a Databricks-based architecture that will support future finance integration capabilities.

Your role:

As Fin Enhance Databricks Engineer, you will help build and scale the strategic Databricks-based platform for global financial data integration. The role combines hands-on Databricks engineering, data architecture, SAP finance integration understanding and practical AI-first ways of working.

The two positions are expected to cover both development and architecture needs: one profile should lean senior/architecture, while the other should be strongly hands‑on in engineering and delivery. Both profiles must be able to work with AI as part of daily delivery, not as a side activity.

Key responsibilities:

• Design, build and maintain Databricks-based pipelines for financial data ingestion, transformation, validation, mapping, aggregation and SAP posting preparation.
• Contribute to architecture standards covering Delta Lake, Unity Catalog, Medallion layers, workflows, environments and reusable deployment patterns.
• Support the migration from legacy integration platforms existing logic and re-architecting it into scalable Databricks/PySpark solutions.
• Use AI-supported methods to accelerate analysis, code generation support, testing, reconciliation, documentation and operational troubleshooting, while keeping human approval and quality gates in place.
• Develop and improve operational applications and dashboards for monitoring, mapping maintenance, reprocessing and support visibility.
• Collaborate with SAP FI, business analysts, architects, developers and production support to ensure solutions are reliable, auditable and fit for finance-critical operations.
• Promote clean engineering practices: version control, reusable components, testing discipline, CI/CD, release traceability and environment promotion from DEV to UAT to PROD.
• Apply AI pragmatically to reduce repetitive manual work in discovery, design, build, test, documentation and support activities.
• Frame problems clearly for AI tools by providing context, constraints, expected output and validation criteria.
• Use AI to identify recurring error patterns, generate draft tests, compare expected vs. actual outcomes, support reconciliation and improve operational insights.
• Maintain human-in-the-loop accountability: AI may accelerate outputs, but the engineer owns validation, controls, implementation quality and production impact.
• Show curiosity, learning agility and adaptability as AI tools evolve; the mindset is as important as experience with a specific AI tool.

Required Knowledge and Skills:

Databricks Platform:

• Strong hands‑on experience with Databricks platform.
• Expertise in Unity Catalog, Workflows, Volumes, Databricks Apps, SQL Warehouses, Delta Lake, and Databricks Asset Bundles.
• Experience working across DEV/UAT/PROD Databricks workspaces.
• Strong proficiency in Python, PySpark, and Spark SQL.
• Hands‑on experience with Medallion Architecture, Auto Loader, ETL/ELT pipelines, data validations, mappings, aggregations, and posting preparation.
• Ability to design and develop scalable, reliable data pipelines.

SAP Finance Integration:

Good understanding of SAP FI concepts, including:

• GL Accounts
• Cost Centers
• Company Codes
• Posting Keys
• Document Types
• Posting Dates
• Reversals
• Experience integrating data pipelines with SAP finance processes is preferred.

AI-First Delivery:

• Ability to use AI tools effectively throughout the engineering lifecycle.
• Code-generation support
• Test creation
• Technical documentation
• Operational troubleshooting

Operational Applications:

• Experience with Dash/Plotly, Dash AG Grid, Dash Bootstrap Components, Flask, Pandas, Databricks SQL Connector, and Databricks SDK.
• Ability to build operational applications and reporting solutions around Databricks data.
• Strong understanding of Git, YAML, CI/CD, version-controlled notebooks, reusable components, code reviews, release promotion, and production support discipline.
• Experience following structured software engineering and deployment practices.

Preferred Experience:

• Experience in insurance or financial services integration, particularly involving GL postings, reversals, reconciliation, and data controls.
• Exposure to heterogeneous source systems or finance/insurance applications.
• Experience with Lakeview Dashboards or Databricks-based operational reporting.
• Experience migrating workloads from Informatica, DB2, or similar legacy ETL platforms to Databricks or other cloud data platforms.
• Databricks certification or equivalent hands‑on implementation experience.