AI Data Engineer

Charterhouse Partnership; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Charterhouse Partnership in Singapore is seeking a Senior AI Data Engineer to lead our data platform transformations as we scale AI initiatives across the enterprise. You will design and operate scalable data ingestion, transformation, and serving solutions, build batch and streaming pipelines, and advance Generative AI workloads with RAG, embeddings, and vector stores.

The role requires hands-on Python/SQL, Spark, cloud data platforms, governance, security, and collaboration with

Qualifications

• 5+ years of experience in Data Engineering, Data Platform Engineering, Cloud Data Engineering.
• Proficient in Python and SQL.
• Experience with Apache Spark/PySpark and large-scale data processing.
• Designing, developing, and operating production-grade batch and/or streaming data pipelines.
• Experience with cloud data platforms such as Databricks, Kafka, Delta Lake, Microsoft Fabric.
• Experience with Generative AI, Retrieval-Augmented Generation (RAG), knowledge bases, embeddings, or vector databases.
• Understanding of data ingestion, transformation, orchestration, data quality, monitoring, and observability.
• Experience integrating data from APIs, databases, files, and other sources.
• Familiarity with CI/CD, Git, DevOps, and DataOps.
• Knowledge of enterprise data governance, security, access control, and compliance.
• Experience supporting production environments, including troubleshooting and performance optimization.
• Strong communication and collaboration skills; ability to lead technical discussions.
• Self-motivated, adaptable, and passionate about emerging data/AI technologies.

Responsibilities

• Design, develop, and operate scalable data ingestion, transformation, and serving solutions across cloud and hybrid environments.
• Build and optimise batch and streaming data pipelines to support analytics, ML, and AI use cases.
• Develop data transformation using Python, SQL, and Apache Spark.
• Integrate data from APIs, databases, files, and streaming platforms.
• Build and operationalise knowledge bases and RAG solutions for Generative AI applications.
• Develop and maintain solutions for knowledge storage, embeddings, vectorisation, and knowledge lifecycle management.
• Ensure data pipelines and AI data solutions are scalable, reliable, secure, observable, and maintainable.
• Monitor and troubleshoot production data workflows, incident triage and root-cause analysis, and drive continuous improvement.
• Contribute to automation, CI/CD, version control, and deployment processes across data and AI platforms.
• Collaborate with engineers, architects, product teams, stakeholders, and source-system owners to translate requirements into effective solutions.
• Provide technical guidance and mentorship to engineers and delivery partners.
• Maintain technical documentation, metadata, data lineage, operational procedures, and solution designs.
• Apply data governance, security, access control, and compliance standards throughout design and implementation.
• Evaluate emerging data/AI technologies and contribute to PoCs and innovative solutions.

Skills

Data Engineering

Python

SQL

Apache Spark

Data Pipelines

Databricks

Kafka

Delta Lake

Microsoft Fabric

Generative AI

RAG

Embeddings

Vector Databases

CI/CD

Git

DevOps

DataOps

Data Governance

Troubleshooting

Communication

Leadership

Adaptability

Education

Bachelor's degree in Computer Science, Engineering, Data Science

Tools

Databricks

Kafka

Delta Lake

Microsoft Fabric

Job description

We're looking for a Senior AI Data Engineer to join a leading Singapore-based enterprise undergoing one of the region's most ambitious AI transformations.

RESPONSIBILITIES:

• Design, develop, and operate scalable data ingestion, transformation, and serving solutions across cloud and hybrid data environments.
• Build and optimise batch and streaming data pipelines to support analytics, machine learning, and AI use cases.
• Develop data transformation and processing solutions using technologies such as Python, SQL, and Apache Spark.
• Integrate data from diverse sources including APIs, databases, files, and streaming platforms.
• Build and operationalise knowledge bases and Retrieval-Augmented Generation (RAG) solutions to support Generative AI and agentic AI applications.
• Develop and maintain solutions for knowledge storage, embeddings, vectorisation, and knowledge lifecycle management.
• Ensure data pipelines and AI data solutions are scalable, reliable, secure, observable, and maintainable.
• Monitor and troubleshoot production data workflows, perform incident triage and root-cause analysis, and drive continuous improvement.
• Contribute to automation, CI/CD, version control, and deployment processes across data and AI platforms.
• Collaborate with engineers, architects, product teams, business stakeholders, and source-system owners to translate requirements into effective technical solutions.
• Provide technical guidance and mentorship to engineers and delivery partners, promoting reusable components, engineering best practices, and production readiness.
• Maintain technical documentation, metadata, data lineage, operational procedures, and solution designs.
• Apply data governance, security, access control, and enterprise compliance standards throughout solution design and implementation.
• Evaluate emerging data and AI technologies and contribute to proof-of-concepts and innovative solutions.

REQUIREMENTS:

• 5+ years of experience in Data Engineering, Data Platform Engineering, Cloud Data Engineering, or a related field.
• Bachelor's degree in Computer Science, Engineering, Data Science, or a related discipline.
• Strong hands-on experience with Python and SQL.
• Experience with Apache Spark/PySpark and large-scale data processing.
• Proven experience designing, developing, and operating production-grade batch and/or streaming data pipelines.
• Experience with modern cloud data platforms and technologies such as Databricks, Kafka, Delta Lake, Microsoft Fabric, or equivalent.
• Experience working with Generative AI, Retrieval-Augmented Generation (RAG), knowledge bases, embeddings, or vector databases.
• Understanding of data ingestion, transformation, orchestration, data quality, monitoring, and observability practices.
• Experience integrating data from APIs, databases, files, and other enterprise data sources.
• Familiarity with CI/CD, Git, DevOps, and DataOps practices.
• Understanding of enterprise data architecture, data governance, security, access control, and compliance requirements.
• Experience supporting production environments, including troubleshooting, incident management, root-cause analysis, and performance optimisation.
• Strong analytical and problem-solving skills with the ability to work through complex technical challenges.
• Excellent written and verbal communication skills, with the ability to collaborate effectively across technical and non-technical stakeholders.
• Ability to work independently, lead technical discussions, and provide guidance to other engineers.
• Self-motivated, adaptable, and passionate about emerging data and AI technologies.