Pretraining Data Engineer – Open Models Platform

Prime Intellect AI; San Francisco (CA); Remote; USD 150,000 - 350,000; Full time

Benefits offered by this job

Flexible work arrangements

Visa sponsorship & relocation

Team off-sites & learning

Job summary

Prime Intellect AI is assembling the open frontier AI platform. We seek a data-focused role to source and curate training data, turning sources into training-ready corpora, and collaborating with researchers to test data efficacy for pretraining and midtraining.

You will build scalable pipelines, ensure data quality, provenance, and reproducibility, and help design data mixtures for long-context reasoning and coding tasks. Remote or SF-office options available.

Qualifications

• Strong Python programming skills and experience building data-processing systems.
• Experience with large datasets and storage/compute tradeoffs.
• Ability to translate research goals into concrete data quality checks.
• Comfort delivering end-to-end data pipelines with researchers.
• Experience with web crawling, document extraction, or corpus curation.

Responsibilities

• Find, evaluate, and collect valuable training data from web, code, and technical sources.
• Build scalable pipelines that convert raw data into training-ready corpora.
• Develop checks for data quality, coverage, and contamination.
• Collaborate with researchers on data mixtures and sampling strategies.
• Prepare data for long-context learning and testing with synthetic data.
• Run experiments and use evaluations to decide data sources and processing.
• Ensure data processing is reproducible with clear provenance and versioning.

Skills

Python programming

Data pipelines

Distributed systems

Data quality checks

Tools

Spark

Ray

Git

Job description

Prime Intellect AI is assembling the open frontier AI platform. We seek a data-focused role to source and curate training data, turning sources into training-ready corpora, and collaborating with researchers to test data efficacy for pretraining and midtraining.

You will build scalable pipelines, ensure data quality, provenance, and reproducibility, and help design data mixtures for long-context reasoning and coding tasks. Remote or SF-office options available.