Member of Technical Staff – Engineering Lead, Data Ingestion

Reflection; New York (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave

Unlimited PTO

Visa sponsorship

Team off-sites

Job summary

Reflection is seeking a Data Ingestion Lead in New York to head the team building the ingestion layer—web crawl, extraction pipelines, and data lakes. You’ll mentor engineers, guide architecture, and coordinate with pre-training, data quality, and partnerships to deliver scalable corpora for AI models.

You’ll stay hands-on, drive technical decisions, and manage day-to-day execution in a fast-moving environment while ensuring auditable, compliant data delivery.

Qualifications

• Experience leading data or infrastructure teams.
• Deep experience building web-scale data acquisition systems.
• Strong coding ability with production-grade pipelines.
• Expertise across crawling, extraction, and data storage domains.
• Familiarity with Ray/Beam/Spark and Airflow/Prefect.
• Understanding of LLM data requirements and experiments.
• Ability to guide strategy and execute across concurrent campaigns.

Responsibilities

• Build, mentor, and grow a high-performing data ingestion team.
• Provide front-line leadership across the ingestion stack: crawl, extraction, and lakes.
• Stay hands-on to contribute as an individual contributor.
• Prioritize work and run data campaigns in a fast-paced environment.
• Guide architectural decisions for scalability, reliability, and cost.
• Collaborate with research, data quality, and partnerships teams.
• Onboard new sources within licensing and robots.txt constraints.

Skills

Data ingestion

Team leadership

Web crawling

Distributed processing

Airflow/Prefect

Spark/Beam

Parquet/JSONL

Dataset versioning

Tools

Ray

Beam

Spark

Airflow

Prefect

Parquet

JSONL

WARC

Job description

Our Mission

Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on. We build open models that let anyone control their intelligence and help shape the future of AI. Our mission: make intelligence open and accessible to all.

About The Role

Reflection’s Data team builds the training corpora our frontier models learn from. Before a model can learn anything, the data has to be found, fetched, extracted, and delivered reliably, responsibly, and at enormous scale. The ingestion layer is the machinery that turns the open web, licensed corpora, and other large-scale sources into well-structured, versioned, auditable datasets for pre-training.

As Data Ingestion Lead, you’ll provide front-line leadership of the team that builds this layer, spanning all three of its pillars web crawl , data ingestion pipelines , and data lakes . You’ll build, mentor, and grow a team of data ingestion engineers, guide the technical and architectural decisions across crawling, extraction, and corpus storage/delivery, and work closely with the pre-training research, data quality, and data partnerships teams that depend on what you ship. You’ll stay close enough to the stack to make targeted contributions as an individual contributor and to maintain a deep understanding of the team’s technical work. Subtle decisions at the ingestion layer what we crawl, how we extract, what we keep ripple through training and directly affect where our models are strong, safe, and where they fail.

What You’ll Do

• Build, mentor, and grow a high-performing team of data ingestion engineers. Coach and support your reports in understanding, and pursuing, their professional growth.
• Provide front-line leadership across the full ingestion stack — web crawling and acquisition, extraction and normalization pipelines, and the data lakes that version and deliver training corpora and the campaigns that run across all three.
• Stay hands-on: become familiar with the team’s technical stack enough to make targeted contributions as an individual contributor.
• Manage day-to-day execution: prioritize the team’s work and run data acquisition and ingestion campaigns in a highly dynamic, fast-paced environment.
• Guide technical and architectural decisions, emphasizing scalability, reliability, auditability, and cost crawler architecture and politeness/scheduling, distributed processing (Ray/Beam/Spark), orchestration (Airflow/Prefect), storage formats and layout (Parquet, JSONL, WARC; object stores and lakehouse formats), deduplication, and dataset versioning and delivery.
• Close the loop with research: partner with pre-training and data quality teams to tie ingestion decisions to measurable downstream model impact, and run experiments to evaluate crawling strategies, extraction methods, and ingestion tradeoffs.
• Work across the wider data effort data partnerships, digitization operations, and external vendors to onboard new sources within legal, licensing, and robots.txt constraints.
• Raise the bar for technical judgment, prioritization, communication, and execution in a fast-moving environment.

What We’re Looking For

• Experience building, mentoring, and growing data or infrastructure engineering teams while staying technically hands-on. (Comfortable growing into leading a larger team quickly if you haven’t managed at that scale before.)
• Deep experience building web-scale data acquisition or ingestion systems, with real ownership of production-grade pipelines at multi-TB to PB scale.
• Strong coding ability and the credibility to earn the technical trust of a strong team.
• Deep expertise in at least one of: web crawling & acquisition, large-scale extraction & ingestion pipelines, or data lakes / corpus storage & delivery with working knowledge across the others, and the ability to learn the rest.
• Fluency with the modern large-scale data toolkit: distributed compute (Ray, Beam, Spark), orchestration (Airflow, Prefect), formats (Parquet, JSONL, WARC), and object-store / data-lake architectures.
• Familiarity with how LLMs are trained and evaluated, and an intuition for what makes data useful for training; comfortable designing experiments and using proxy quality signals to guide system improvements.
• Ability to guide strategy and drive execution across concurrent data campaigns, and to partner effectively with research teams, external data vendors, and partnerships/legal on acquisition.

What We Offer

We believe that to make intelligence open and accessible to all, you need to start at the foundation. Joining Reflection means building from the ground up as part of a talent-dense team. You will help define our future as a company, and help define the future of open foundational models.

We want you to do the most impactful work of your career with the confidence that you and the people you care about most are supported.

• Top-tier compensation: Salary and equity structured to recognize and retain our talent globally.
• Stock options: Everyone who joins and contributes to Reflection’s success gets to share in the upside through stock options.
• Health & wellness: Comprehensive medical, dental, vision, and life, with an annual wellness allowance.
• Meals: Lunch and dinner are provided in the office daily.
• Life & family: 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys.
• Vacation days: Unlimited paid time off in the U.S. and 30 days in the U.K.
• Sponsorship support: We sponsor visas to help exceptional talent join our team and support long-term immigration pathways where applicable.
• Team building: We have regular off-sites, happy hours, and team celebrations.

Export Control Notice: This position may require access to technology or source code subject to the U.S. Export Administration Regulations. Any offer of employment for this role may be conditioned on the Company’s ability to provide the candidate with access to such technology or source code in compliance with applicable U.S. export control laws, which may require the Company to seek government authorization.