Senior Data Engineer-1

Prudential plc; Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Prudential plc seeks a data engineer to design, build, and maintain batch and near-real-time data pipelines on Prudential Singapore’s UDP platform. You will deliver auditable data products for analytics, AI, and commissions calculations, collaborating with business, finance, technology, data, and platform teams.

You will implement data models, quality checks, and scalable processing while ensuring security and governance across pipelines.

Qualifications

• Hands-on experience with Python, SQL, Databricks, Spark and data pipeline development.
• Experience designing ETL/ELT solutions, data models, transformations, and workflow orchestration.
• Experience implementing complex calculation logic, reconciliation, and financial or commissions-related data processing.
• Good understanding of data quality, observability, lineage, metadata, data contracts, and access controls.
• Experience with Delta Lake, Unity Catalog, Git, automated testing, CI/CD, and cloud data platforms.
• Understanding of APIs, file-based integration, event processing, and secure data exchange.
• Good knowledge of Agile delivery, software engineering practices, and production support.
• Strong problem analysis, troubleshooting, communication, and collaboration skills.
• Insurance or financial services experience is an advantage.

Responsibilities

• Design, build, test, deploy, and maintain batch and near-real-time data pipelines on the UDP platform.
• Develop data ingestion, transformation, enrichment, reconciliation, and publishing processes.
• Build and maintain data processing for commissions calculations, including rules, rates, adjustments, incentives, deductions, and calculation outputs.
• Implement reusable data models and curated datasets aligned with enterprise and UDP standards.
• Ensure data accuracy, completeness, lineage, traceability, security, and auditability across pipelines and calculation processes.
• Implement automated data quality checks, controls, alerts, and exception handling.
• Optimize pipeline performance, reliability, scalability, and cost.
• Use source control, automated testing, CI/CD, environment controls, and deployment standards.
• Monitor production pipelines, investigate failures and data discrepancies, and provide operational support.
• Contribute to the enhancement and maintenance of core UDP platform capabilities, engineering standards, and reusable components.

Skills

Python

SQL

Databricks

Spark

ETL/ELT

Data Modeling

CI/CD

Cloud

Education

Bachelor in Computer Science

Tools

Delta Lake

Unity Catalog

Git

CI/CD tools

Job description

Prudential’s purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people’s career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.

In this role, you will design, build, maintain, and support data pipelines and core data capabilities on Prudential Singapore’s Unified Data Platform (UDP). You will deliver reliable and auditable data products supporting business operations, analytics, AI, and commissions calculations. As part of this dynamic role, you will work closely with business, finance, technology, data, and platform teams to transform source data into trusted and reusable datasets across the organization.

Job Description

• Design, build, test, deploy, and maintain batch and near-real-time data pipelines on the UDP platform.
• Develop data ingestion, transformation, enrichment, reconciliation, and publishing processes.
• Build and maintain data processing for commissions calculations, including rules, rates, adjustments, incentives, deductions, and calculation outputs.
• Implement reusable data models and curated datasets aligned with enterprise and UDP standards.
• Ensure data accuracy, completeness, lineage, traceability, security, and auditability across pipelines and calculation processes.
• Implement automated data quality checks, controls, alerts, and exception handling.
• Optimize pipeline performance, reliability, scalability, and cost.
• Use source control, automated testing, CI/CD, environment controls, and deployment standards.
• Monitor production pipelines, investigate failures and data discrepancies, and provide operational support.
• Contribute to the enhancement and maintenance of core UDP platform capabilities, engineering standards, and reusable components.

Who we are looking for

• Strong hands-on experience with Python, SQL, Databricks, Spark, and data pipeline development.
• Experience designing ETL/ELT solutions, data models, transformations, and workflow orchestration.
• Experience implementing complex calculation logic, reconciliation, and financial or commissions-related data processing.
• Good understanding of data quality, observability, lineage, metadata, data contracts, and access controls.
• Experience with Delta Lake, Unity Catalog, Git, automated testing, CI/CD, and cloud data platforms.
• Understanding of APIs, file-based integration, event processing, and secure data exchange.
• Good knowledge of Agile delivery, software engineering practices, and production support.
• Strong problem analysis, troubleshooting, communication, and collaboration skills.
• Insurance or financial services experience is an advantage.

Working Experience

• At least 5 years of experience in data engineering or software engineering.
• Hands-on experience delivering and supporting production data pipelines and enterprise data platforms.

Education

• Bachelor in Computer Science, Data Engineering, Information Systems, Engineering or equivalent.

Language

• Fluent written and spoken English.

Prudential is an equal opportunity employer. We provide equality of opportunity of benefits for all who apply and who perform work for our organisation irrespective of sex, race, age, ethnic origin, educational, social and cultural background, marital status, pregnancy and maternity, religion or belief, disability or part-time / fixed-term work, or any other status protected by applicable law. We encourage the same standards from our recruitment and third-party suppliers taking into account the context of grade, job and location. We also allow for reasonable adjustments to support people with individual physical or mental health requirements.