Senior Data Engineer

Zof AI; San Francisco, Northern (CA, KY); Hybrid; USD 140,000 - 200,000; Full time

Job summary

Zof AI in San Francisco, CA is seeking a Senior Data Engineer to build robust pipelines that convert raw agent runs, test results, and remediation outcomes into trustworthy data. You will own the data layer beneath our control plane—handling streaming and batch ingestion, warehouse models, and quality checks to keep datasets defensible.

The role emphasizes data quality as an engineering problem, with daily dependencies on pipelines used across evals and analytics.

Qualifications

• Experience building and operating production data pipelines.
• Strong SQL and data modeling in a warehouse.
• Python or similar language for data work.
• Experience with orchestration tools (Airflow, Dagster, Prefect).
• Understanding of data quality, testing, and schema evolution.
• Judgment about freshness, cost, and correctness trade-offs.
• Clear written and verbal communication.
• High ownership of the systems you build.

Responsibilities

• Design and build ingestion pipelines for agent runs, defect reproductions, and remediation outcomes.
• Model and maintain the warehouse tables that evals, analytics, and product reporting depend on.
• Build streaming and batch paths that keep data fresh without sacrificing correctness.
• Own pipeline orchestration end to end, including scheduling, dependencies, retries, and backfills.
• Wire data quality checks, tests, and alerting into every pipeline you ship.
• Track lineage so every number we report traces back to its source evidence.
• Partner with AI and product engineers to turn raw signals into datasets they can build on.
• Own the cost, performance, and reliability of the data platform in production.

Skills

SQL proficiency

Data modeling

Python

Data quality mindset

Ownership

Communication

Tools

Airflow

Dagster

Prefect

dbt

Snowflake

BigQuery

Databricks

Job description

Zof AI is seeking a Senior Data Engineer to build the pipelines that turn raw agent runs, test results, and remediation outcomes into data the company can trust. This role owns the data layer beneath our control plane: streaming and batch ingestion, the warehouse models that serve evals and analytics, and the quality checks and lineage that make every dataset defensible. If you have worked as an Analytics Engineer, ETL Developer, Big Data Engineer, or Data Platform Engineer, this is that discipline at Zof AI. The ideal candidate has built pipelines that other teams depend on daily and treats data quality as an engineering problem, not a cleanup task.

Engineering · Mid to Senior · Full-time · On-site · San Francisco, CA

Responsibilities

• Design and build ingestion pipelines for agent runs, defect reproductions, and remediation outcomes.
• Model and maintain the warehouse tables that evals, analytics, and product reporting depend on.
• Build streaming and batch paths that keep data fresh without sacrificing correctness.
• Own pipeline orchestration end to end, including scheduling, dependencies, retries, and backfills.
• Wire data quality checks, tests, and alerting into every pipeline you ship.
• Track lineage so every number we report traces back to its source evidence.
• Partner with AI and product engineers to turn raw signals into datasets they can build on.
• Own the cost, performance, and reliability of the data platform in production.

Requirements

• Experience building and operating production data pipelines.
• Strong SQL and comfort modeling data in a warehouse.
• Proficiency in Python or a similar language for data work.
• Working knowledge of an orchestration tool such as Airflow, Dagster, or Prefect.
• Understanding of data quality, testing, and schema evolution.
• Judgment about freshness, cost, and correctness trade-offs.
• Clear written and verbal communication.
• High ownership of the systems you build.

Nice to have

• Experience with streaming systems such as Kafka, Kinesis, or Pub/Sub.
• Experience with dbt, Snowflake, BigQuery, Databricks, or similar tooling.
• Experience preparing datasets for evals, fine-tuning, or model training.
• Experience standing up a data platform at an early-stage company.

Hands-on experience building data pipelines that feed AI, ML, or eval workloads is required