Lead Data Acquisition & Crawling Platform Architect

NVIDIA Corporation; Massachusetts; Híbrido; USD 272.000 - 431.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Equity

Benefits

Descripción de la vacante

NVIDIA is seeking an engineering lead to architect and scale the data acquisition infrastructure that powers our foundation models. You will own the architecture of a unified platform for source discovery, crawling, and ingestion, guiding teams across multiple disciplines.

You’ll drive design reviews, mentor engineers, and build scalable, compliant pipelines for open-web, licensed, and partner data; with emphasis on throughput, quality, and reproducibility for training data.

Formación

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience.
• 15+ years of experience in software engineering, including designing and operating large-scale distributed systems.
• Deep experience with web crawling or related discovery, indexing, or ingestion infrastructure.
• Hands-on engineering and system design skills across architecture, implementation, performance analysis, and production debugging.

Responsabilidades

• Design scalable frontier storage and queueing systems with high-throughput writes and efficient lookups.
• Build a continuous discovery loop that extracts links and feeds new candidates into the crawl frontier.
• Develop a shared crawl execution layer that fetches reliably and politely at scale.
• Create extensible pipelines for open-web, licensed, and partner content across multimodal assets.
• Define interfaces for source connectors to enable adding sources without rebuilding the platform.
• Partner with parsing and curation teams to preserve code, math, and structured content.
• Implement provenance and policy enforcement with governance partners.
• Own production readiness, capacity planning, observability, fault tolerance, and cost efficiency.
• Instrument the pipeline end-to-end and set objectives for throughput, freshness, and training cutoffs.
• Collaborate with research/product teams to evaluate crawls and datasets through pilots.

Conocimientos

Web crawling

Distributed systems

Data ingestion

System design

Leadership

Educación

BS, MS, or PhD in Computer Science/Computer Engineering or related technical field

Descripción del empleo

NVIDIA is seeking an engineering lead to architect and scale the data acquisition infrastructure that powers our foundation models. You will own the architecture of a unified platform for source discovery, crawling, and ingestion, guiding teams across multiple disciplines.

You’ll drive design reviews, mentor engineers, and build scalable, compliant pipelines for open-web, licensed, and partner data; with emphasis on throughput, quality, and reproducibility for training data.