Data Pipelines Lead for AI-Driven Finance

Autonomous Technologies Group; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

ATG, Autonomous Technologies Group, is building a data lake and AI research infrastructure for financial markets. You will own the data platform end to end, from ingestion and storage to transformation, cataloging, governance, and access for research and production use.

You will unify diverse data sources, handle schema changes and backfills, ensure point‑in‑time correctness and provenance, and enable AI-native workflows with secure, well‑described datasets and APIs for researchers and agents.

Qualifications

• Strong Python and SQL, plus distributed processing experience.
• Production ownership of data systems at scale.
• Temporal data, provenance, and data governance.
• AI-native workflows and safe data systems.
• Familiarity with financial data is a plus.

Responsibilities

• Own the data platform end to end: ingestion, storage, transformation, cataloging, governance, and access across research and production.
• Build reliable pipelines: onboard large, heterogeneous datasets; handle schema changes, backfills and vendor failures.
• Guarantee research-grade correctness: point-in-time data, lineage, versioning, reproducibility.
• Create trusted data products: normalize identifiers, timestamps, reference data, and unstructured sources into accessible datasets and APIs.
• Enable AI-native systems: build permission-aware interfaces and datasets for researchers and agents.
• Partner with research and engineering to turn ambiguous problems into durable data capabilities.

Skills

Python

SQL

Distributed processing

Data engineering

Research data experience

Tools

Airflow

Spark

Cloud platforms

Job description

ATG, Autonomous Technologies Group, is building a data lake and AI research infrastructure for financial markets. You will own the data platform end to end, from ingestion and storage to transformation, cataloging, governance, and access for research and production use.

You will unify diverse data sources, handle schema changes and backfills, ensure point‑in‑time correctness and provenance, and enable AI-native workflows with secure, well‑described datasets and APIs for researchers and agents.