Data Ingestion Lead for Large-Scale AI Training

Reflection AI Ltd; New York (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals in office

Life & family

Unlimited vacation

Visa sponsorship

Team off-sites

Job summary

Reflection AI Ltd. is seeking a Data Ingestion Lead to head a team that builds the ingestion stack for pre-training data. You’ll steer web crawl, extraction pipelines, and data lakes, balancing scale, reliability, and cost while mentoring engineers and collaborating with research and partnerships.

You’ll remain hands-on to understand the stack and guide architectural decisions, working in a fast-paced environment to deliver high-quality training corpora for frontier models.

Qualifications

• Experience leading data or infrastructure teams while remaining hands-on.
• Built web-scale data acquisition or ingestion systems with production pipelines.
• Strong coding abilities and earning trust of a team.
• Expertise in web crawling, ingestion pipelines, or data lakes with cross-domain knowledge.
• Proficiency with Ray, Beam, Spark; Airflow or Prefect; Parquet/JSONL/WARC; object stores.
• Familiar with data for training large language models and experimental design.
• Ability to drive strategy across concurrent campaigns with research and partnerships teams.

Responsibilities

• Build, mentor, and grow a high-performing team.
• Provide front-line leadership across the full ingestion stack.
• Stay hands-on and contribute as an individual contributor.
• Manage day-to-day execution of data campaigns in a fast-paced environment.
• Guide technical and architectural decisions for scalability and cost.
• Collaborate with research, data quality, and partnerships teams.
• Coordinate with external vendors and data sources within licensing rules.

Skills

Team leadership

Web-scale data ingestion

Hands-on coding

Data lake / storage

Distributed compute tooling

Experiment design for data quality

Tools

Ray

Beam

Spark

Airflow

Prefect

Parquet

JSONL

WARC

Job description

Reflection AI Ltd. is seeking a Data Ingestion Lead to head a team that builds the ingestion stack for pre-training data. You’ll steer web crawl, extraction pipelines, and data lakes, balancing scale, reliability, and cost while mentoring engineers and collaborating with research and partnerships.

You’ll remain hands-on to understand the stack and guide architectural decisions, working in a fast-paced environment to deliver high-quality training corpora for frontier models.