Data Infrastructure Engineer — Build Clean, Trusted Datasets

Economic Intelligence Co.; San Francisco, Northern (CA, KY); Hybrid; USD 140,000 - 210,000; Full time

Job summary

Economic Intelligence Co. in San Francisco (SF preferred) is seeking a Software Engineer, Data Infrastructure to build the data backbone for our models, evals, and incubation teams.

You will own batch and streaming pipelines, storage and lineage, and tools for researchers to query and trust the data.

A strong background in Python and SQL with Spark, Kafka, Airflow and dbt will help turn messy traces into clean, reliable datasets.

Qualifications

• Strong Python and SQL with experience in Spark, Ray, Kafka, Flink, Airflow, dbt or similar.
• Strong instincts for data quality, schema design, performance and cost.
• Built systems where correctness, lineage and reliability mattered.
• Able to turn messy events and traces into clean datasets without losing context.
• Comfortable working across infra, research, product, and operations.
• Notices when something is off before anyone else does.

Responsibilities

• 01 Batch and streaming pipelines for training, post-training, evaluation, and product data.
• 02 Storage, processing, lineage, deduplication, quality checks, and dataset versioning for company-building traces.
• 03 Data systems that connect agent activity to outcomes: demand, customers, delivery, revenue, and verification.
• 04 Tools researchers and engineers use to query, inspect, debug, and trust the data.

Skills

Python

SQL

Spark

Kafka

Flink

Airflow

dbt

Data quality

Schema design

Reliability

Tools

Spark

Kafka

Flink

Airflow

dbt

Ray

Job description

Economic Intelligence Co. in San Francisco (SF preferred) is seeking a Software Engineer, Data Infrastructure to build the data backbone for our models, evals, and incubation teams.

You will own batch and streaming pipelines, storage and lineage, and tools for researchers to query and trust the data.

A strong background in Python and SQL with Spark, Kafka, Airflow and dbt will help turn messy traces into clean, reliable datasets.