Software Engineer, Data Infrastructure San Francisco or New York

Eico; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

Economic Intelligence Co. in San Francisco (SF preferred) is seeking a Software Engineer, Data Infrastructure to build the data backbone for our models, evals, and incubation teams.

You will own batch and streaming pipelines, storage and lineage, and tools for researchers to query and trust the data.

A strong background in Python and SQL with Spark, Kafka, Airflow and dbt will help turn messy traces into clean, reliable datasets.

Qualifications

• Strong Python and SQL with experience in Spark, Ray, Kafka, Flink, Airflow, dbt or similar.
• Strong instincts for data quality, schema design, performance and cost.
• Built systems where correctness, lineage and reliability mattered.
• Able to turn messy events and traces into clean datasets without losing context.
• Comfortable working across infra, research, product, and operations.
• Notices when something is off before anyone else does.

Responsibilities

• 01 Batch and streaming pipelines for training, post-training, evaluation, and product data.
• 02 Storage, processing, lineage, deduplication, quality checks, and dataset versioning for company-building traces.
• 03 Data systems that connect agent activity to outcomes: demand, customers, delivery, revenue, and verification.
• 04 Tools researchers and engineers use to query, inspect, debug, and trust the data.

Skills

Python

SQL

Spark

Kafka

Flink

Airflow

dbt

Data quality

Schema design

Reliability

Tools

Spark

Kafka

Flink

Airflow

dbt

Ray

Job description

Engineering · Company Building Agents San Francisco or New York · SF preferred

Software Engineer, Data Infrastructure

You will build the data backbone for Eico.

Our models, evals, and incubation teams depend on clean traces of real company-building work: what agents tried, what customers did, what was delivered, what converted, and what was verified. You'll turn that raw activity into trustworthy datasets for training, post-training, evaluation, and product.

What you'll own:

• 01 Batch and streaming pipelines for training, post-training, evaluation, and product data.
• 02 Storage, processing, lineage, deduplication, quality checks, and dataset versioning for company-building traces.
• 03 Data systems that connect agent activity to outcomes: demand, customers, delivery, revenue, and verification.
• 04 Tools researchers and engineers use to query, inspect, debug, and trust the data.

What we're looking for:

• Strong Python and SQL. Spark, Ray, Kafka, Flink, Airflow, dbt, or similar experience preferred.
• Strong instincts for data quality, schema design, performance, and cost.
• Has built systems where correctness, lineage, and reliability mattered.
• Can turn messy events and traces into clean datasets without losing important context.
• Comfortable working across infra, research, product, and operations.
• Notices when something is off before anyone else does.