Research Data Engineer, Pretraining — Remote & Equity

Prime Intellect, Inc.; San Francisco (CA); Hybrid; USD 150,000 - 350,000; Full time

Benefits offered by this job

Cash Compensation

Remote or on-site

Relocation support

Team off-sites

Learning opportunities

Job summary

Prime Intellect, Inc. in San Francisco is seeking a data-focused engineer to build training data pipelines for open frontier AI models.

You will source valuable data, convert it into training-ready corpora, and collaborate with researchers to test what actually improves model performance. Responsibilities include data extraction from web and code sources, deduplication, quality checks, and scalable processing with clear provenance and dataset versioning.

Qualifications

• Strong Python skills and a track record of building reliable data-processing or distributed systems.
• Experience handling large datasets with attention to storage, compute, throughput, and reproducibility.
• Ability to diagnose data-quality issues and translate research goals into measurable checks and experiments.
• Comfort owning an evolving technical problem end-to-end and collaborating with researchers to test assumptions.

Responsibilities

• Find, evaluate, and collect valuable training data from web, code, and technical sources.
• Build scalable pipelines that turn raw data into training-ready corpora through extraction, cleaning, deduplication, and filtering.
• Develop checks for data quality, coverage, and contamination, and use them to improve collection and processing.
• Work with researchers on data mixtures and sampling strategies for pretraining and midtraining.
• Prepare and evaluate data for long-context learning, reasoning, and coding, testing synthetic data where it proves useful.
• Run controlled data experiments and use model evaluations to decide which sources, mixtures, and processing choices are worth scaling.
• Make large-scale data processing efficient, reproducible, and recoverable, with clear source provenance and dataset versioning.

Skills

Python

Data pipelines

Data quality checks

Research collaboration

Tools

Spark

Ray

Job description

Prime Intellect, Inc. in San Francisco is seeking a data-focused engineer to build training data pipelines for open frontier AI models.

You will source valuable data, convert it into training-ready corpora, and collaborate with researchers to test what actually improves model performance. Responsibilities include data extraction from web and code sources, deduplication, quality checks, and scalable processing with clear provenance and dataset versioning.