Data Ingestion Engineer for Large-Scale AI Datasets

Reflection AI Ltd; New York (NY); On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave

Unlimited vacation

Visa sponsorship

Team off-sites

Job summary

Reflection AI Ltd. is expanding its Data Team to build and operate large-scale data ingestion systems for pre-training frontier models. You will own ingestion pipelines, crawlers, and data delivery, collaborating with researchers to optimize model performance.

The role blends research and engineering, emphasizing experimentation, data-driven improvements, and robust, observable systems.

Qualifications

• Experience building web crawling, data ingestion, or large-scale data acquisition systems using Ray, Beam, Spark, or similar technologies.
• Familiarity with how LLMs are trained and evaluated, and an intuition for what makes data useful for training
• Comfortable working with very large datasets (multi-TB to PB scale) and building systems that are observable, testable, and maintainable
• Comfortable designing experiments and using data to guide system improvements
• Excellent communication skills. You can explain system behavior. You consider and communicate tradeoffs clearly

Responsibilities

• Build and operate large-scale data ingestion systems for pre-training, including web crawling, extraction, and dataset delivery
• Run experiments to evaluate crawling strategies, extraction methods, and ingestion tradeoffs
• Analyze ingested data to identify gaps, redundancy, and areas to improve
• Build ingestion pipelines that scale reliably across large data campaigns
• Develop specialized crawlers for high-priority data sources
• Review code, debug production issues, and continuously improve ingestion infrastructure

Skills

Web crawling

Data ingestion

Distributed data processing

Experiment design

Communication

Tools

Ray

Beam

Spark

Job description

Reflection AI Ltd. is expanding its Data Team to build and operate large-scale data ingestion systems for pre-training frontier models. You will own ingestion pipelines, crawlers, and data delivery, collaborating with researchers to optimize model performance.

The role blends research and engineering, emphasizing experimentation, data-driven improvements, and robust, observable systems.