Staff Engineer - AI Data Engineering

Prudential (Cambodia) Life Assurance; Singapore; On-site; SGD 180,000 - 300,000; Full time

Job summary

Prudential (Cambodia) Life Assurance seeks an experienced AI Data Engineering Lead to define and lead the engineering of enterprise-scale data platforms powering AI, ML, GenAI and analytics. You will build an AI-ready data foundation by unifying data engineering, cloud platforms, and governance.

You will guide architecture, implement scalable pipelines, and ensure data quality, security, and lineage across diverse data sources for enterprise AI initiatives.

Responsibilities

• Define the enterprise AI Data Engineering strategy, architecture and engineering standards.
• Design scalable data architectures supporting AI/ML, GenAI, LLM, analytics and enterprise data products.
• Lead engineering of data platforms providing trusted data for ML models, LLMs and GenAI applications.
• Build ingestion and transformation frameworks for data from databases, APIs, documents, events, logs and enterprise apps.
• Drive automated data validation, quality controls, metadata management and lineage.

Skills

AI data engineering

Azure

Python

SQL

Spark/PySpark

Databricks

Kafka

dbt

Tools

Azure Databricks

Azure Data Factory

Azure Synapse

Snowflake

Databricks

Airflow

MLflow

Job description

Prudential's purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people's career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.

We are looking for an experienced AI Data Engineering Lead to define and lead the engineering of enterprise-scale data platforms and data products that power Artificial Intelligence, Machine Learning, Generative AI and advanced analytics.

This individual will build the organisation's AI-ready data foundation by bringing together modern data engineering, cloud platforms, batch and real-time processing, unstructured data, vector and semantic technologies, data governance, and AI/ML engineering practices.

The role requires a senior technology leader with 15+ years of experience, strong hands-on technical depth, architecture capability, and a proven ability to lead engineering teams and deliver complex data platforms at enterprise scale.

Key Responsibilities

AI Data Engineering Strategy & Architecture

• Define the enterprise AI Data Engineering strategy, architecture and engineering standards.
• Design scalable data architectures supporting AI/ML, GenAI, LLM, analytics and enterprise data products.
• Establish reusable architectural patterns for structured, semi-structured and unstructured data.
• Define patterns for data lakes, lakehouse architectures, data warehouses, streaming platforms, vector databases and AI knowledge layers.
• Drive architectural decisions covering performance, scalability, resiliency, security and cost.

AI-Ready Data Platforms

• Lead engineering of data platforms providing trusted and reusable data for ML models, LLMs, AI agents and GenAI applications.
• Build ingestion and transformation frameworks for data from databases, APIs, documents, events, logs and enterprise applications.
• Establish pipelines for batch, near-real-time and streaming data processing.
• Drive automated data validation, quality controls, metadata management and lineage.
• Establish engineering approaches for preparing text, documents, images and other unstructured data for AI consumption.

GenAI & LLM Data Engineering

• Provide technical leadership for Retrieval-Augmented Generation (RAG), document ingestion, parsing, chunking and enrichment.
• Lead embedding generation and management, vector and hybrid search, semantic retrieval and reranking.
• Design knowledge repositories, metadata and access-control filtering, and enterprise search integration.
• Create data pipelines for LLM evaluation, AI grounding, authorised retrieval, fine-tuning where applicable, and AI agents.
• Understand how poor retrieval, incomplete or stale data, weak metadata and inappropriate chunking affect enterprise AI reliability.

Modern Data Engineering

• Provide leadership across ETL/ELT, distributed processing, lakehouse and data warehouse architectures, streaming, event-driven design, data APIs and data services.
• Drive data modelling, data contracts, observability, quality engineering, lineage and metadata management.
• Embed Infrastructure-as-Code, CI/CD, DevOps and DataOps practices in the engineering lifecycle.
• Improve reusability, automation, reliability and developer productivity through common frameworks and standards.

Cloud & Technology Platforms

• Demonstrate significant expertise across modern cloud and data platforms such as Microsoft Azure, Azure Databricks, Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Snowflake and Databricks.
• Bring strong capability in Python, SQL, Apache Spark/PySpark, Kafka, Delta Lake, Airflow and dbt.
• Apply AI and GenAI data technologies including Azure AI Search, vector databases, embedding models, semantic search, RAG frameworks, MLflow and feature stores.
• Experience with Azure OpenAI, enterprise LLM platforms and agent architectures is highly valuable. Deep expertise is expected across a relevant enterprise-scale stack, rather than every technology listed.

Data Governance, Security & Responsible AI

• Embed data governance, privacy, security and regulatory controls into data engineering architecture.
• Implement appropriate access controls for sensitive and confidential information.
• Ensure lineage, traceability and auditability across AI data pipelines.
• Ensure AI solutions retrieve and process only appropriately authorised data.
• Partner with Cybersecurity, Risk, Compliance, Data Governance and AI Governance teams, while promoting Responsible AI engineering practices.
• Experience handling PII, sensitive customer information, data residency and regulatory requirements is highly valuable.

Data Quality & Observability

• Define enterprise standards for AI data quality and automated