Web-Scale Data Systems Engineer

Visa Hunt; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 230,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals

Life & family

Vacation days

Sponsorship support

Team building

Job summary

Reflection is a research lab building open models and tools for everyone. We are seeking an engineer to own and operate web-scale crawling systems that discover, acquire, and process content from across the internet.

You will manage URL discovery, scheduling, distributed crawling, and data delivery at scale, collaborating with researchers to optimize coverage and data quality. You will work with world-class researchers to determine which parts of the web matter most for model performance and

Qualifications

• Experience building large‑scale web crawling, search indexing, or internet‑scale data collection systems.
• Strong understanding of crawling architectures, URL frontier management, scheduling, and distributed crawl coordination.
• Experience with Ray, Spark, Beam, Flink or similar frameworks for large data processing.
• Familiarity with content extraction, HTML parsing, browser automation, rendering systems and modern web technologies.
• Experience operating systems that process petabyte-scale datasets.
• Strong systems engineering skills including reliability, observability, performance optimization and debugging.
• Experience designing experiments and using data to improve crawl quality, coverage, and efficiency.
• Excellent communication and ability to reason about system tradeoffs and constraints.

Responsibilities

• Build and operate web-scale crawling infrastructure capable of collecting data across billions of URLs.
• Design and optimize URL discovery, prioritization, scheduling, and crawl orchestration systems.
• Develop distributed crawlers that acquire content while respecting site constraints and operational needs.
• Build systems for content extraction, rendering, parsing, and normalization across diverse web formats.
• Improve crawl coverage, freshness, efficiency, and quality through measurement and experimentation.
• Design infrastructure for large-scale recrawling, change detection, and incremental updates.
• Develop specialized crawlers for high-value domains and dynamic websites.
• Analyze crawl performance and web coverage to identify gaps and opportunities.
• Build observability, monitoring, and reliability systems for large-scale crawl operations.
• Debug production issues and continuously improve performance, scalability, and resilience.

Skills

Web crawling

Distributed systems

URL frontier management

Scheduling

Content extraction

HTML parsing

Rendering systems

Browser automation

Petabyte-scale datasets

Education

Bachelor's degree in CS or related

Tools

Ray

Spark

Beam

Flink

Job description

Reflection is a research lab building open models and tools for everyone. We are seeking an engineer to own and operate web-scale crawling systems that discover, acquire, and process content from across the internet.

You will manage URL discovery, scheduling, distributed crawling, and data delivery at scale, collaborating with researchers to optimize coverage and data quality. You will work with world-class researchers to determine which parts of the web matter most for model performance and