Lead Data Acquisition Architect (Crawl & Ingest)

NVIDIA Corporation; Colorado; On-site; USD 272,000 - 431,000; Full time

Benefits offered by this job

Equity and benefits

Job summary

NVIDIA is seeking an engineering lead to architect and scale data acquisition infrastructure powering our foundation models. You will own the unified platform for source discovery, URL indexing, crawl frontiers, and ingestion, guiding teams through architecture reviews and critical implementation work.

This hands-on leadership role spans distributed systems, crawl infrastructure, and third-party data ingestion, with a focus on production readiness, capacity planning, and observable reliability.

Qualifications

• BS/MS/PhD in Computer Science, Computer Engineering, or related field, or equivalent experience.
• 15+ years of experience in software engineering, including designing and operating large-scale distributed systems across multiple teams.
• Deep experience with web crawling, discovery, indexing, or ingestion infrastructure, including distributed scheduling, stateful services, storage, and high-throughput data processing.
• Strong hands-on engineering and system design skills; able to move between architecture, implementation, performance analysis, and production debugging.
• Experience building batch or streaming pipelines and reasoning about consistency, retries, backpressure, and recovery; improving throughput and reliability.
• Experience delivering data platforms for research, ML, or demanding technical users; translating requirements into clear interfaces and measurable outcomes.
• Strong technical communication and leadership through influence across multiple teams.

Responsibilities

• Design scalable frontier storage and queueing systems for high-throughput writes and efficient lookups.
• Develop a shared crawl execution layer that fetches reliably at scale and handles dynamic websites.
• Build extensible pipelines for open-web, licensed, and partner content with stable interfaces for sources.
• Collaborate with parsing and curation teams to preserve valuable content and improve data yield.
• Implement provenance, policy enforcement, and auditable data governance controls.
• Own production readiness: capacity planning, observability, fault tolerance, and cost efficiency; set throughput and reliability targets.
• Work with researchers and product teams to evaluate crawls and external datasets through pilots; deliver reproducible data releases.
• Coordinate dependencies across teams and balance short-term delivery with long-term platform development.

Skills

Distributed systems

System design

Web crawling

Batch/streaming pipelines

Production debugging

Data platforms for ML/Research

Technical leadership

Cross-functional collaboration

Education

BS/MS/PhD in CS/CE or equivalent

Job description

NVIDIA is seeking an engineering lead to architect and scale data acquisition infrastructure powering our foundation models. You will own the unified platform for source discovery, URL indexing, crawl frontiers, and ingestion, guiding teams through architecture reviews and critical implementation work.

This hands-on leadership role spans distributed systems, crawl infrastructure, and third-party data ingestion, with a focus on production readiness, capacity planning, and observable reliability.