Member of Technical Staff | Knowledge Platform

Avra

United States

Remote

USD 120,000 - 180,000

Full time

43 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Avra is building scalable data systems to collect, process, and prepare information for research, model training, and production predictions. The role covers data acquisition from diverse sources—including non-API portals—plus streaming, batch, and lakehouse infrastructure to create reliable, versioned relational datasets.

You will own end-to-end data pipelines, from crawling and scraping to transformation with Spark, dbt, and SQL, ensuring data quality, freshness, and accessibility for

Qualifications

  • Experience building collection systems for difficult sources and production data pipelines at scale.
  • Ability to work with event-driven ingestion and batch or incremental data processing.
  • Willingness to take operational responsibility and respond when systems fail.
  • Daily use of coding agents, paired with sound judgment about safe use.

Responsibilities

  • Build and operate crawlers, scrapers, and browser automation, with scheduling, retries, session handling, and change detection.
  • Monitor source coverage and freshness, identify changes in site layouts or access, and restore collection when it breaks.
  • Run event-driven ingestion using Kafka and Pub/Sub, handling ordering, replay, dead-letter queues, backpressure, and idempotent processing.
  • Develop batch and incremental data pipelines with Spark, dbt, and SQL to produce clean, typed relational tables.
  • Own lakehouse storage built around Parquet and support dependable datasets for downstream research and production use.
  • Operate the systems you build, including responding to incidents affecting sources or pipelines.

Skills

Data pipelines
Event-driven ingestion
Batch processing
Operational mindset

Tools

Kafka
Pub/Sub
Spark
dbt
SQL
Parquet

Job description

Role overview

Own the data systems that collect, process, and prepare information for research, model training, and production predictions. The work spans acquisition from websites and portals—including sources without APIs—and the streaming, batch, and lakehouse infrastructure that turns raw captures into reliable, versioned relational datasets.

Responsibilities
  • Build and operate crawlers, scrapers, and browser automation, with scheduling, retries, session handling, and change detection.
  • Monitor source coverage and freshness, identify changes in site layouts or access, and restore collection when it breaks.
  • Run event-driven ingestion using Kafka and Pub/Sub, handling ordering, replay, dead-letter queues, backpressure, and idempotent processing.
  • Develop batch and incremental data pipelines with Spark, dbt, and SQL to produce clean, typed relational tables.
  • Own lakehouse storage built around Parquet and support dependable datasets for downstream research and production use.
  • Operate the systems you build, including responding to incidents affecting sources or pipelines.
Requirements
  • Practical experience building collection systems for difficult sources and production data pipelines at scale.
  • Ability to work with event-driven ingestion and batch or incremental data processing.
  • Willingness to take operational responsibility and respond when systems fail.
  • Daily use of coding agents, paired with sound judgment about safe use.
Nice to have
  • Experience with JavaScript-heavy sites, authenticated sessions, rate limits, proxy management, or tools such as Playwright, Puppeteer, and Scrapy.
  • Workflow orchestration using Airflow, Dagster, or Temporal.
  • Entity resolution and relational or graph data modeling across multiple sources.
  • ML dataset practices such as point-in-time correctness, reproducible training sets, and evaluation splits.
  • Familiarity with cloud data services on GCP or AWS, OpenTelemetry, or financial and regulated environments.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal Software Engineer, Platform Services (PMTS)
Principal Software Engineer, Platform Services (PMTS)

Leadout Capital • San Francisco (CA)

On-site
USD 190,000 - 270,000
MTS - Engineering (Data Infrastructure)
MTS - Engineering (Data Infrastructure)

Collinear AI, Inc. • San Francisco (CA)

On-site
USD 180,000 - 240,000
Member of Technical Staff — Data Infrastructure
Member of Technical Staff — Data Infrastructure

Kindredventures • San Francisco (CA)

On-site
USD 130,000 - 170,000
Fundable - Senior Data Engineer
Fundable - Senior Data Engineer

silver • United States

Remote
USD 140,000 - 180,000
Senior / Staff Data Platform Engineer (US)
Senior / Staff Data Platform Engineer (US)

Craft.co • United States

Hybrid
USD 153,000 - 187,000
Equity
Unlimited vacation
Health, Dental, Vision insurance
+1
Staff Software Engineer, Data Ingestion
Staff Software Engineer, Data Ingestion

Remote • San Francisco (CA)

On-site
USD 180,000 - 280,000
Member of Technical Staff — Data Ingestion & Quality
Member of Technical Staff — Data Ingestion & Quality

Kindredventures • San Francisco (CA)

On-site
USD 120,000 - 180,000
Data Platform Engineer
Data Platform Engineer

BrightAI • Palo Alto (CA)

On-site
USD 150,000 - 210,000
Member of Technical Staff, Software Engineer (Data Infrastructure)
Member of Technical Staff, Software Engineer (Data Infrastructure)

Orbifold AI, Inc. • Palo Alto (CA)

On-site
USD 140,000 - 200,000
Member of Technical Staff, Software Engineering
Member of Technical Staff, Software Engineering

Touchdown Labs • United States

Remote
USD 120,000 - 160,000