Senior Data Engineer: Architect Trusted Data Pipelines

Zof AI; San Francisco, Northern (CA, KY); Hybrid; USD 140,000 - 200,000; Full time

Job summary

Zof AI in San Francisco, CA is seeking a Senior Data Engineer to build robust pipelines that convert raw agent runs, test results, and remediation outcomes into trustworthy data. You will own the data layer beneath our control plane—handling streaming and batch ingestion, warehouse models, and quality checks to keep datasets defensible.

The role emphasizes data quality as an engineering problem, with daily dependencies on pipelines used across evals and analytics.

Qualifications

• Experience building and operating production data pipelines.
• Strong SQL and data modeling in a warehouse.
• Python or similar language for data work.
• Experience with orchestration tools (Airflow, Dagster, Prefect).
• Understanding of data quality, testing, and schema evolution.
• Judgment about freshness, cost, and correctness trade-offs.
• Clear written and verbal communication.
• High ownership of the systems you build.

Responsibilities

• Design and build ingestion pipelines for agent runs, defect reproductions, and remediation outcomes.
• Model and maintain the warehouse tables that evals, analytics, and product reporting depend on.
• Build streaming and batch paths that keep data fresh without sacrificing correctness.
• Own pipeline orchestration end to end, including scheduling, dependencies, retries, and backfills.
• Wire data quality checks, tests, and alerting into every pipeline you ship.
• Track lineage so every number we report traces back to its source evidence.
• Partner with AI and product engineers to turn raw signals into datasets they can build on.
• Own the cost, performance, and reliability of the data platform in production.

Skills

SQL proficiency

Data modeling

Python

Data quality mindset

Ownership

Communication

Tools

Airflow

Dagster

Prefect

dbt

Snowflake

BigQuery

Databricks

Job description

Zof AI in San Francisco, CA is seeking a Senior Data Engineer to build robust pipelines that convert raw agent runs, test results, and remediation outcomes into trustworthy data. You will own the data layer beneath our control plane—handling streaming and batch ingestion, warehouse models, and quality checks to keep datasets defensible.

The role emphasizes data quality as an engineering problem, with daily dependencies on pipelines used across evals and analytics.