Data Engineer (AI Native, Pretrain Algorithm, data pipeline, production)

DADACONSULTANTS PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

DADACONSULTANTS PTE. LTD. is seeking a Data Engineer to own end-to-end data pipelines for large-scale model training in a cutting-edge AI research environment.

You will process web-scale data, structure diverse data types, and implement filtering and deduplication systems. Collaboration with LLM researchers and ML engineers is essential for improving pre-training data quality.

Qualifications

• Strong software engineering fundamentals with expert-level Python proficiency.
• Hands-on experience with distributed processing frameworks (Spark, Ray, Flink, Beam).
• Proven experience designing and data cleaning pipelines, with cost/throughput trade-offs.
• NLP/LLM data concepts including tokenisation, deduplication, text extraction and quality filtering.
• Experience building pre-training or mid-training datasets from scratch (bonus).

Responsibilities

• Build and scale data pipelines processing billions of documents for large-scale model training.
• Process and structure web and document data, including text, tables, formulas, and code.
• Develop data filtering, quality evaluation, and deduplication systems using rules, ML models, and LLMs.
• Optimize large-scale data processing for performance, scalability, and cost efficiency.
• Run experiments to evaluate the impact of data strategies on model performance.
• Collaborate with LLM researchers and ML engineers to continuously improve pre-training data quality.

Skills

Python

Data pipelines

Data cleaning

NLP/LLM data concepts

LLM data pretraining

Tools

Spark

Ray

Flink

Beam

Job description

About the Role

Our client is a well-funded AI research company at the frontier of large language model development, building foundational models that power next-generation AI applications. As the team scales its pre-training infrastructure, they are looking for a skilled Data Engineer to own the end-to-end data pipeline — from raw web-scale ingestion through to high-quality tokenized training datasets. This is a high-impact, technically deep role sitting at the intersection of distributed systems engineering and LLM research, and is ideal for engineers who care deeply about data quality and scale.

Key Responsibilities

• Build and scale data pipelines processing billions of documents for large-scale model training
• Process and structure web and document data, including text, tables, formulas, and code
• Develop data filtering, quality evaluation, and deduplication systems using rules, ML models, and LLMs
• Optimize large-scale data processing for performance, scalability, and cost efficiency
• Run experiments to evaluate the impact of data strategies on model performance
• Collaborate with LLM researchers and ML engineers to continuously improve pre-training data quality

Requirements

• Strong software engineering fundamentals with expert-level Python proficiency and a track record of production-quality code
• Hands-on experience with at least one distributed processing framework such as Spark, Ray, Flink, or Beam
• Proven experience designing and data cleaning pipelines, with a solid understanding of cost and throughput trade-offs
• Deep familiarity with NLP/LLM data concepts including tokenisation, deduplication, text extraction, and quality filtering
• Experience building pre-training or mid-training datasets from scratch (is a bonus)