Senior Pretraining Data Engineer — Scalable ML Pipelines (Remote)

Poolside; Emea; On-site; IDR 1,575,630,252 - 2,275,910,364; Full time

Benefits offered by this job

Fully remote work & flexible hours

37 days/year of vacation & holidays

Health insurance allowance for you & dependents

Company-provided equipment

Well-being, always-be-learning & home office allowances

Frequent team get togethers

Diverse & inclusive people-first culture

Job summary

Poolside is seeking a core member of the Pretraining Data team to build and scale its Model Factory. This hands-on position will involve architecting and maintaining high-performance pipelines that transform vast amounts of raw data into high-quality datasets needed for the company’s models. Candidates should have experience in distributed data systems for machine learning, strong Python skills, and a passion for building innovative solutions in a remote work environment.

Qualifications

• Strong background in building production-grade, distributed data systems.
• Expert-level python knowledge and ability to write clean and maintainable code.
• Strong algorithmic foundations.

Responsibilities

• Build and maintain high-performance pipelines for trillions of tokens.
• Deliver diverse and high-quality datasets for pre-training foundation models.
• Collaborate closely with other teams to ensure alignment on model quality.

Skills

Distributed data systems for machine learning

Python programming

Orchestration (Slurm, Airflow, Dagster)

Observability & Reliability (CI/CD, Grafana, Prometheus)

Experience with libraries like Polars, Dask, or PySpark

Tools

Docker

k8s

Cloud managed services

Job description

Poolside is seeking a core member of the Pretraining Data team to build and scale its Model Factory. This hands-on position will involve architecting and maintaining high-performance pipelines that transform vast amounts of raw data into high-quality datasets needed for the company’s models. Candidates should have experience in distributed data systems for machine learning, strong Python skills, and a passion for building innovative solutions in a remote work environment.