Mission-Driven Backend Engineer: Data Pipelines & Scrapers

Vannevar; United States; Remote; USD 120,000 - 165,000; Full time

Benefits offered by this job

Health insurance

Remote-friendly with WeWork access

Unlimited PTO and year-end downtime

401(k) match

Wellbeing stipend

Military reserve duty pay top-up

Parental leave

Travel care reimbursement

Job summary

Vannevar builds AI systems for national security missions. As part of the Collection Platform team, you’ll own web scrapers, data pipelines, and third-party integrations to deliver complete, timely data even with unreliable upstream systems.

You’ll partner with mission teams, product managers, and engineers to balance data quality, reliability, scale, and cost while shaping system architecture and standards.

Qualifications

• 4+ years of backend engineering experience with production web scraping or data ingestion.
• Proficient in Python and relational databases; solid API design.
• Strong focus on data quality across large datasets.
• Experience designing systems tolerant of upstream failures and changing interfaces.

Responsibilities

• Build and operate production web scrapers, data pipelines, and integrations.
• Design for schema changes, rate limits, authentication, and partial failures.
• Implement retry, backfill, deduplication, idempotency, and replay mechanisms.
• Define data completeness, correctness, freshness, duplication, and provenance metrics.
• Develop APIs, databases, and shared services for collection workflows.
• Collaborate with mission teams, product managers, and engineers to evaluate sources and new data requirements.
• Influence architecture and engineering standards for collection systems.

Skills

Backend engineering

Python

API design

Data quality

Reliability & scalability

Communication

Tools

PostgreSQL

REST APIs

Job description

Vannevar builds AI systems for national security missions. As part of the Collection Platform team, you’ll own web scrapers, data pipelines, and third-party integrations to deliver complete, timely data even with unreliable upstream systems.

You’ll partner with mission teams, product managers, and engineers to balance data quality, reliability, scale, and cost while shaping system architecture and standards.