Senior AI Data Engineer for Scalable AI Pipelines

Charterhouse Partnership; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Charterhouse Partnership in Singapore is seeking a Senior AI Data Engineer to lead our data platform transformations as we scale AI initiatives across the enterprise. You will design and operate scalable data ingestion, transformation, and serving solutions, build batch and streaming pipelines, and advance Generative AI workloads with RAG, embeddings, and vector stores.

The role requires hands-on Python/SQL, Spark, cloud data platforms, governance, security, and collaboration with

Qualifications

• 5+ years of experience in Data Engineering, Data Platform Engineering, Cloud Data Engineering.
• Proficient in Python and SQL.
• Experience with Apache Spark/PySpark and large-scale data processing.
• Designing, developing, and operating production-grade batch and/or streaming data pipelines.
• Experience with cloud data platforms such as Databricks, Kafka, Delta Lake, Microsoft Fabric.
• Experience with Generative AI, Retrieval-Augmented Generation (RAG), knowledge bases, embeddings, or vector databases.
• Understanding of data ingestion, transformation, orchestration, data quality, monitoring, and observability.
• Experience integrating data from APIs, databases, files, and other sources.
• Familiarity with CI/CD, Git, DevOps, and DataOps.
• Knowledge of enterprise data governance, security, access control, and compliance.
• Experience supporting production environments, including troubleshooting and performance optimization.
• Strong communication and collaboration skills; ability to lead technical discussions.
• Self-motivated, adaptable, and passionate about emerging data/AI technologies.

Responsibilities

• Design, develop, and operate scalable data ingestion, transformation, and serving solutions across cloud and hybrid environments.
• Build and optimise batch and streaming data pipelines to support analytics, ML, and AI use cases.
• Develop data transformation using Python, SQL, and Apache Spark.
• Integrate data from APIs, databases, files, and streaming platforms.
• Build and operationalise knowledge bases and RAG solutions for Generative AI applications.
• Develop and maintain solutions for knowledge storage, embeddings, vectorisation, and knowledge lifecycle management.
• Ensure data pipelines and AI data solutions are scalable, reliable, secure, observable, and maintainable.
• Monitor and troubleshoot production data workflows, incident triage and root-cause analysis, and drive continuous improvement.
• Contribute to automation, CI/CD, version control, and deployment processes across data and AI platforms.
• Collaborate with engineers, architects, product teams, stakeholders, and source-system owners to translate requirements into effective solutions.
• Provide technical guidance and mentorship to engineers and delivery partners.
• Maintain technical documentation, metadata, data lineage, operational procedures, and solution designs.
• Apply data governance, security, access control, and compliance standards throughout design and implementation.
• Evaluate emerging data/AI technologies and contribute to PoCs and innovative solutions.

Skills

Data Engineering

Python

SQL

Apache Spark

Data Pipelines

Databricks

Kafka

Delta Lake

Microsoft Fabric

Generative AI

RAG

Embeddings

Vector Databases

CI/CD

Git

DevOps

DataOps

Data Governance

Troubleshooting

Communication

Leadership

Adaptability

Education

Bachelor's degree in Computer Science, Engineering, Data Science

Tools

Databricks

Kafka

Delta Lake

Microsoft Fabric

Job description

Charterhouse Partnership in Singapore is seeking a Senior AI Data Engineer to lead our data platform transformations as we scale AI initiatives across the enterprise. You will design and operate scalable data ingestion, transformation, and serving solutions, build batch and streaming pipelines, and advance Generative AI workloads with RAG, embeddings, and vector stores.

The role requires hands-on Python/SQL, Spark, cloud data platforms, governance, security, and collaboration with