Staff Data Pipelines Engineer for AI-Driven Finance

Autonomous Technologies Group; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

ATG (Autonomous Technologies Group) in San Francisco is seeking a data-focused engineer to design, build, and maintain a scalable data lake and associated pipelines for AI research and production systems.

You will own ingestion, storage, transformation, governance, and access, collaborating with researchers and engineers to ensure accurate, versioned, and queryable data for humans and agents.

Qualifications

• Expert in Python and SQL with production-grade data systems.
• Experience designing end-to-end data platforms at scale.
• Knowledge of temporal data and point-in-time queries.
• Strong data governance, provenance, and privacy practices.
• Familiarity with AI-native data workflows and research data.

Responsibilities

• Own the data platform end to end: ingestion, storage, transformation, cataloging, governance, and access across research and production.
• Build reliable pipelines for large, heterogeneous datasets with schema changes, late data, revisions, backfills, and vendor failures.
• Guarantee research-grade correctness: point-in-time data, lineage, versioning, and reproducibility; prevent look-ahead and survivorship bias.
• Create trusted data products: normalize identifiers, timestamps, corporate actions, reference data, and unstructured sources into accessible datasets and APIs.
• Enable AI-native systems: build permission‑aware interfaces, metadata, datasets, and benchmarks designed for both researchers and agents.
• Partner across the team: work directly with research and engineering to turn ambiguous problems into durable data capabilities.

Skills

Python

SQL

Distributed processing

Data engineering

Production ownership

Temporal data

Data provenance

Privacy

AI-native workflows

Financial data experience

Job description

ATG (Autonomous Technologies Group) in San Francisco is seeking a data-focused engineer to design, build, and maintain a scalable data lake and associated pipelines for AI research and production systems.

You will own ingestion, storage, transformation, governance, and access, collaborating with researchers and engineers to ensure accurate, versioned, and queryable data for humans and agents.