Senior API & Data Connectors Engineer - High-Throughput Data

Three Pillars Recruiting; San Francisco (CA); Sur place; USD 180 000 - 230 000; Plein temps

Résumé du poste

Three Pillars Recruiting is seeking a Senior Software Engineer for the Data Products team in San Francisco. You’ll own and extend the core data ingestion framework and SDK abstractions, designing and maintaining the connector layers at the heart of the data pipeline.

The role centers on building high-throughput, memory-efficient connectors and a robust Flask runtime service. You will work with DuckDB, Arrow, Iceberg, and S3, integrating PyAirbyte adapters and enabling hundreds of data sources to

Qualifications

• 5+ years writing production software in Python.
• Experience building APIs with Flask or similar frameworks for high-throughput JSON/stream data.
• Proven track record in extensible SDKs or framework architectures.

Responsabilités

• Architect Connector Abstractions with clean, typed interfaces.
• Build high-performance native connectors for local filesystems and cloud storage.
• Own and optimize the Flask runtime service for catalog, check, discover, read operations.
• Integrate and potentially extend PyAirbyte adapters to access external data sources.
• Optimize data transfer semantics for low latency and memory efficiency.
• Ensure developers can easily author, test, and deploy connectors with strong typing.

Connaissances

Python

Async/Await

Typed Python

API design

SDK design

Data ingestion

High-performance computing

Outils

Flask

DuckDB

Apache Arrow

Iceberg

S3

PyAirbyte

Poetry

Docker

Description du poste

Three Pillars Recruiting is seeking a Senior Software Engineer for the Data Products team in San Francisco. You’ll own and extend the core data ingestion framework and SDK abstractions, designing and maintaining the connector layers at the heart of the data pipeline.

The role centers on building high-throughput, memory-efficient connectors and a robust Flask runtime service. You will work with DuckDB, Arrow, Iceberg, and S3, integrating PyAirbyte adapters and enabling hundreds of data sources to