Data Ingestion Engineer for Scalable AI Pipelines

Visa Hunt; San Francisco, Northern (CA, KY); Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Stock options

Health & wellness

Meals provided

Parental leave

Job summary

Reflection is seeking an engineer to build and operate ingestion systems that turn open web and large-scale data sources into reliable corpora for training frontier models. You will own data acquisition, normalization, versioning, and delivery to pre-training pipelines, collaborating with researchers and data quality teams.

You will design and run experiments, analyze data quality, and develop scalable, observable ingestion infrastructure across campaigns.

Qualifications

• Experience building web crawling, data ingestion, or large-scale data acquisition systems using Ray, Beam, Spark, or similar technologies.
• Familiarity with how LLMs are trained and evaluated, and an intuition for what makes data useful for training.
• Comfortable working with very large datasets and building observable, testable, maintainable systems.

Responsibilities

• Build and operate large-scale data ingestion systems for pre-training, including web crawling, extraction, and dataset delivery.
• Run experiments to evaluate crawling strategies, extraction methods, and ingestion tradeoffs.
• Analyze ingested data to identify gaps, redundancy, and areas to improve.
• Build ingestion pipelines that scale reliably across large data campaigns.
• Develop specialized crawlers for high-priority data sources.
• Review code, debug production issues, and continuously improve ingestion infrastructure.

Skills

Experiment design

Cross-functional collaboration

Communication skills

Tools

Ray

Beam

Spark

Job description

Reflection is seeking an engineer to build and operate ingestion systems that turn open web and large-scale data sources into reliable corpora for training frontier models. You will own data acquisition, normalization, versioning, and delivery to pre-training pipelines, collaborating with researchers and data quality teams.

You will design and run experiments, analyze data quality, and develop scalable, observable ingestion infrastructure across campaigns.