Research Engineer, Training Systems & Experimentation - Monee

Monee; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Monee is seeking a Research Engineer to bridge model research and engineering, building reliable experimentation and training foundations for behavioral sequence pretraining and graph learning. You will tackle bottlenecks in data loading, sampling, computation, memory, and evaluation, while improving turnaround, resource use, and observability for researchers.

You will contribute to reproducible workflows, performance baselines, and efficient sequence training across multi-GPU settings,

Qualifications

• Bachelor's degree or above in Computer Science, Software Engineering, AI, or a related field with strong software foundations and demonstrable research engineering, ML engineering, or systems projects.
• Proficiency in Python and PyTorch with understanding of training loops, autograd, batches, optimizers, checkpoints, and inference, and the ability to debug model code.
• In-depth experience in training systems, data pipelines, performance diagnosis, distributed systems, or experimentation infrastructure with clear bottlenecks, trade-offs, and improvements.
• Foundational performance analysis skills across GPUs, CPUs, memory, and I/O, using profilers and runtime metrics to identify issues.
• Attention to model quality, experimental comparability, numerical correctness, data versions, and reproducibility for engineering changes.
• Strong practices in code organization, testing, logging, recovery, and documentation, with ability to work with researchers on open-ended problems.
• Ability to read technical material, write engineering docs, and collaborate across teams.

Responsibilities

• Build reproducible experimentation workflows for training, checkpointing, recovery, evaluation, and result recording; track code, data, graph snapshots, sampling, and randomness.
• Profile data loading, sampling, transfer, forward/backward passes, synchronization, and output writing; establish benchmarks for throughput, memory, and turnaround to target quality.
• Improve sequence training efficiency through data sharding, prefetching, bucketing, padding, batching, mixed precision, and checkpointing; evaluate bottlenecks across backends.
• Advance graph learning engineering by optimizing storage, neighbor sampling, feature fetching, caching, CPU–GPU transfer, and training pipelines with fixed snapshots for repeatable evals.
• Develop evaluation and diagnostic tools linking training, embedding generation, downstream integration, and reporting; expose losses, gradients, anomalies, and statistics.
• Estimate capacity and compute budgets, compare single-run vs parallel experiments, and improve recovery and execution efficiency.
• Support reliable integration with application and platform teams; maintain benchmarks, tools, runbooks, and reusable code.

Skills

Python

PyTorch

ML engineering

System design

Education

Bachelor's degree in CS

Tools

DGL

PyG

CUDA

Job description

Research Engineer, Training Systems & Experimentation - Monee

Experienced (Individual Contributor) Singapore

About Monee

Monee is a leading digital payments and financial services provider in Southeast Asia, with a growing presence in Latin America. Its mission is to better the lives of individuals and businesses in the region with financial services through technology.

Monee delivers a comprehensive range of digital financial products and services, including mobile wallets, payment processing, credit, banking, and insurtech. By leveraging technology and strategic partnerships, Monee aims to drive greater digital financial inclusion, convenience, and accessibility for individuals and small businesses across Southeast Asia and Latin America.

Monee is part of Sea Limited, a global technology company.

About the Team

We are looking for a Research Engineer who bridges model research and engineering to build reliable, efficient, and reproducible experimentation and training foundations for behavioral sequence pretraining and graph learning, reducing the time needed to reach trustworthy research conclusions.

You will work with algorithm experts, researchers, and application owners on real training workloads, addressing bottlenecks in data loading, graph sampling, computation, memory, experiment execution, and evaluation. You will improve experiment turnaround, resource use, and batch representation generation while preserving model quality and data correctness, and build observability tools that support model diagnosis.

Job Description

• Reproducible experimentation: Build configurable workflows for training, checkpointing, recovery, evaluation, and result recording. Track code, data, graph snapshots, sampling, and randomness, reducing duplicate scripts and manual work.
• Performance baselines and diagnosis: Profile loading, sampling, transfer, forward and backward passes, synchronization, and output writing. Establish benchmarks for throughput, memory, failure rate, turnaround, and GPU-hours required to reach a target quality.
• Sequence training efficiency: Improve data sharding, prefetching, length bucketing, padding, batching, mixed precision, and checkpointing. Evaluate attention backends, compilation, or multi-GPU approaches against measured bottlenecks while checking numerical and training correctness.
• Graph learning engineering: Optimize graph storage, neighbor sampling, feature fetching, caching, CPU–GPU transfer, and training pipelines. Support repeatable evaluation on fixed snapshots and validate sampling or graph-construction changes with researchers.
• Evaluation and diagnostic tools: Connect training, embedding generation, downstream integration, and reporting. Expose losses, gradients, numerical anomalies, and representation statistics to help researchers diagnose issues, while preserving point-in-time correctness of features and relationships.
• Resource efficiency and experiment scheduling: Estimate capacity and compute budgets from measured throughput, memory, and concurrency needs. Compare scaling a single run with running independent experiments in parallel and improve recovery and execution efficiency.
• Inference and engineering delivery: Optimize batch representation or score generation, sharding, outputs, and versioning. Support reliable integration with application and platform teams and maintain benchmarks, tools, runbooks, and reusable code.

What You Will Work On

• Build research engineering capabilities around real sequence and graph learning problems and directly improve experiment speed and trustworthiness.
• Investigate bottlenecks across models, data, and systems and validate improvements through quality and cost measurements.
• Work with researchers and application colleagues to build engineering foundations the team can reuse over time.

Requirements

• Bachelor's degree or above in Computer Science, Software Engineering, AI, or a related field, with strong software engineering foundations and demonstrable research engineering, ML engineering, or systems projects.
• Proficiency in Python and PyTorch or comparable frameworks, with an understanding of training loops, automatic differentiation, batches, optimizers, checkpoints, and inference, and the ability to debug model code independently.
• In-depth practical experience in at least one of training systems, data pipelines, performance diagnosis, distributed systems, or experimentation infrastructure, with clear evidence of bottlenecks, trade-offs, and improvements.
• Foundational performance analysis skills across GPUs, CPUs, memory, and I/O, with the ability to use profilers and runtime metrics to identify issues rather than relying solely on trial and error.
• Attention to model quality, experimental comparability, numerical correctness, data versions, and reproducibility, with the ability to design fair quality and cost comparisons for engineering changes.
• Strong practices in code organization, testing, logging, recovery, and documentation, with the ability to work through open-ended problems with researchers and deliver iteratively.
• Ability to read technical material, write engineering documentation, and collaborate across teams.

Good to have

• Experience with Transformer pretraining or fine-tuning, GNNs, or large-scale behavioral data.
• Experience with DGL or PyG, graph sampling, sparse computation, feature stores, or large-graph training optimization.
• Experience with multi-GPU training, DDP or FSDP, mixed precision, or distributed data processing, with an understanding of scaling efficiency and cost trade-offs.
• Experience with CUDA or Triton, kernel optimization, compilation, experiment orchestration, evaluation platforms, or observability tooling.