Data Ingestion Engineering Lead: Web Crawling & Pipelines

Reflection; New York (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave

Unlimited PTO

Visa sponsorship

Team off-sites

Job summary

Reflection is seeking a Data Ingestion Lead in New York to head the team building the ingestion layer—web crawl, extraction pipelines, and data lakes. You’ll mentor engineers, guide architecture, and coordinate with pre-training, data quality, and partnerships to deliver scalable corpora for AI models.

You’ll stay hands-on, drive technical decisions, and manage day-to-day execution in a fast-moving environment while ensuring auditable, compliant data delivery.

Qualifications

• Experience leading data or infrastructure teams.
• Deep experience building web-scale data acquisition systems.
• Strong coding ability with production-grade pipelines.
• Expertise across crawling, extraction, and data storage domains.
• Familiarity with Ray/Beam/Spark and Airflow/Prefect.
• Understanding of LLM data requirements and experiments.
• Ability to guide strategy and execute across concurrent campaigns.

Responsibilities

• Build, mentor, and grow a high-performing data ingestion team.
• Provide front-line leadership across the ingestion stack: crawl, extraction, and lakes.
• Stay hands-on to contribute as an individual contributor.
• Prioritize work and run data campaigns in a fast-paced environment.
• Guide architectural decisions for scalability, reliability, and cost.
• Collaborate with research, data quality, and partnerships teams.
• Onboard new sources within licensing and robots.txt constraints.

Skills

Data ingestion

Team leadership

Web crawling

Distributed processing

Airflow/Prefect

Spark/Beam

Parquet/JSONL

Dataset versioning

Tools

Ray

Beam

Spark

Airflow

Prefect

Parquet

JSONL

WARC

Job description

Reflection is seeking a Data Ingestion Lead in New York to head the team building the ingestion layer—web crawl, extraction pipelines, and data lakes. You’ll mentor engineers, guide architecture, and coordinate with pre-training, data quality, and partnerships to deliver scalable corpora for AI models.

You’ll stay hands-on, drive technical decisions, and manage day-to-day execution in a fast-moving environment while ensuring auditable, compliant data delivery.