Senior Data Acquisition Architect – Web Crawling & Ingestion

NVIDIA Corporation; Boston (MA); On-site; USD 272,000 - 431,000; Full time

Job summary

NVIDIA seeks an engineering lead to architect and scale the data acquisition infrastructure powering foundation models. You will start with code and STEM data for Nemotron, building systems that crawl the open web at scale and deliver high-quality data to training pipelines.

This role spans distributed systems, crawl infrastructure, and third-party data ingestion, with ownership of the unified platform architecture and end-to-end production readiness.

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or related field, or equivalent experience.
• 15+ years of software engineering experience designing and operating large-scale distributed systems.
• Deep experience with web crawling or closely related discovery, indexing, or ingestion infrastructure, including distributed scheduling, stateful services, storage, and high-throughput data processing.
• Strong hands-on engineering and system design skills. Ability to move between architecture, implementation, performance analysis, and production debugging.
• Experience building batch or streaming pipelines and reasoning about consistency, retries, duplicate processing, backpressure, and recovery.
• Experience delivering data platforms for research, ML, search, or other demanding technical users.
• Strong technical communication and leadership through influence across engineering, product, research, and governance teams.

Responsibilities

• Design scalable frontier storage and queueing systems with high-throughput writes and efficient lookups.
• Develop a shared crawl execution layer that fetches reliably at scale.
• Build extensible pipelines for open-web, licensed, and partner content.
• Collaborate with parsing and curation teams to preserve valuable code and content.
• Implement provenance and policy enforcement with governance partners.
• Own production readiness, including capacity planning, observability, fault tolerance, and cost efficiency.
• Evaluate internal crawls and external datasets through pilots; deliver reproducible data releases.
• Coordinate dependencies across teams and balance immediate delivery with long-term platform development.

Skills

Web crawling

Distributed systems

System design

Technical leadership

Engineering mentoring

Education

BS/MS/PhD in CS or related field

Job description

NVIDIA seeks an engineering lead to architect and scale the data acquisition infrastructure powering foundation models. You will start with code and STEM data for Nemotron, building systems that crawl the open web at scale and deliver high-quality data to training pipelines.

This role spans distributed systems, crawl infrastructure, and third-party data ingestion, with ownership of the unified platform architecture and end-to-end production readiness.