Lead Databricks Engineer [T500-29957]

Talent500; Hyderabad; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Talent500 is hiring for one of its clients to build scalable data pipelines on Databricks, focusing on data quality, security, and governance for financial platforms.

The role requires around 5 years of data engineering experience, strong PySpark/SQL skills, Delta Lake expertise, and cloud experience with Azure or AWS, working with cross-functional teams in a fast-paced environment.

Qualifications

• Approximately 5 years of experience in data engineering, analytics engineering, or a related technology role.
• Hands-on experience developing and implementing solutions on Databricks within enterprise environments.

Responsibilities

• Design, develop, and maintain data pipelines and analytical workloads using Databricks.
• Develop data transformations using PySpark, Spark SQL, and Delta Lake.
• Support end-to-end data flows across ingestion, curation, and analytics layers.
• Contribute to the development and continuous improvement of Databricks frameworks, standards, and best practices.

Skills

Python

SQL

PySpark

Data pipelines

Data governance

Cloud platforms

CI/CD

Analytical thinking

Education

Bachelor's degree in Computer Science or related

Tools

Databricks

Delta Lake

Unity Catalog

Git

Azure

AWS

Job description

Talent500 is hiring for one of its clients.

Who are we:

Core Insurance Platforms (CIP) is Zurich’s global capability responsible for building, running, and evolving core insurance technology. We set a unified, scalable operating model—covering governance, standards, architecture, service delivery, and reuse—so our business units can deliver at speed and scale.

CIP is the strategic steward of Zurich’s Guidewire ecosystem, aligning platform roadmaps to business strategy while driving stability, modernization, reduced supplier dependency, and long term cost efficiency.

India delivery center is one of our global delivery and capability hub. We bring together experts in AI, engineering, analysis, quality, and architecture to deliver product & process solutions, application run services, change and transformation initiatives, and centralized platform services across both on prem and Guidewire Cloud environments. Our teams operate from multiple global delivery centers, supporting Zurich’s business units worldwide.

Role Purpose:

The Databricks Engineer will design, develop, and maintain scalable, secure, and audit-ready data pipelines and analytical workloads on Databricks. The role will support financial platforms by enabling reliable data ingestion, transformation, calculation, and reporting, with a strong focus on data quality, performance, security, and governance.

Key Accountabilities:

• Design, develop, and maintain data pipelines and analytical workloads using Databricks.
• Develop data transformations using PySpark, Spark SQL, and Delta Lake.
• Support end-to-end data flows across ingestion, curation, and analytics layers.
• Contribute to the development and continuous improvement of Databricks frameworks, standards, and best practices.

Data Quality, Performance & Reliability:

• Implement robust data validation, logging, monitoring, and error-handling mechanisms.
• Optimize workloads for performance, scalability, reliability, and cost efficiency.
• Ensure data pipelines are reliable, recoverable, and observable.
• Identify and resolve data quality and pipeline performance issues.

Security, Governance & Audit:

• Implement and operate data governance controls using Unity Catalog, including access control, lineage, and auditability.
• Ensure solutions comply with internal controls, segregation-of-duties requirements, and audit standards.
• Maintain appropriate traceability of data and processing results to support regulatory, audit, and business reviews.
• Follow established security and governance standards when designing and implementing data solutions.
• Work closely with actuarial SMEs, testers, architects, and other stakeholders to translate business requirements into effective technical solutions.
• Support SIT, UAT, and production release activities by resolving data, performance, and pipeline-related issues.
• Participate in technical reviews and contribute to solution documentation and knowledge sharing.
• Collaborate with cross-functional teams to continuously improve data engineering practices and platform capabilities.

Skills & Experience:

Required Experience:

• Approximately 5 years of experience in data engineering, analytics engineering, or a related technology role.
• Hands-on experience developing and implementing solutions on Databricks within enterprise environments.

Technical Skills:

• Strong programming experience in Python/PySpark and SQL.
• Hands-on experience with Delta Lake, data modelling, and data transformation patterns.
• Experience designing, developing, and operating data pipelines in cloud environments such as Azure or AWS.
• Familiarity with version control, CI/CD, and software engineering best practices.
• Understanding of data engineering principles, including data ingestion, transformation, validation, and optimization.

Databricks & Platform Knowledge:

• Hands-on experience with Databricks development and platform capabilities.
• Experience with Unity Catalog or similar data governance frameworks is an advantage.
• Good understanding of performance tuning, cluster configuration, workload optimization, and cost management.
• Exposure to actuarial, financial, risk, or insurance-related data domains is an advantage but not mandatory.

Professional Skills:

• Strong analytical and problem-solving skills.
• Structured, detail-oriented, and quality-focused approach to engineering.
• Ability to troubleshoot complex data and pipeline issues effectively.
• Strong communication and collaboration skills, with the ability to work with both technical and non-technical stakeholders.
• Open, learning-oriented mindset with a willingness to continuously improve technical skills and engineering practices.

Education:

• Bachelor’s degree in Computer Science, Engineering, Mathematics, Data Science, or a related field, or equivalent practical experience.

Ideal Candidate Profile:

This role is suited to an engineer who enjoys hands-on Databricks development, values data quality, security, and governance, and is passionate about building reliable, scalable data platforms that support critical actuarial and financial processes.