Senior Data Engineer — Pipelines for AI‑Driven Infrastructure

Foresight Works; New York (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health

Dental

Vision

401(k)

PTO

Job summary

Foresight is seeking a Senior Data Engineer to build the data foundation behind our platform. You will design and maintain pipelines that ingest project data from customers’ scheduling tools and turn it into clean, well‑modeled data that powers our predictions, analytics and customer reporting.

This hands‑on role focuses on data quality, observability, and governance. You’ll collaborate with ML engineers and full‑stack teams, design for failure, and optimize cost as data volumes grow—shaping

Qualifications

• 5+ years of experience building and running production data pipelines, with ownership from source to consumer.
• Strong in Python and advanced SQL, including writing and tuning complex queries.
• Deep data modeling experience in Postgres or a similar relational database: schema design, indexing, partitioning and migrations.
• Experience parsing and normalizing messy real-world data: large files, inconsistent formats, missing or contradictory values.
• Hands-on with workflow orchestration for multi-step pipelines with dependencies, retries and backfills.
• Experience with AWS data services such as S3, managed databases, streaming and serverless compute.
• Testing mindset for data: validation, quality checks, and reproducible, idempotent pipelines.
• Clear in communication and comfortable making pragmatic trade-offs on a small team.

Responsibilities

• Own pipelines from design through production: define data contracts, set freshness and quality targets, and keep them met.
• Partner with ML engineers on the features and datasets their models need, and with full stack engineers on the data behind each product feature.
• Design for failure: pipelines that retry safely, can be re-run without duplicating data, and recover cleanly from bad inputs.
• Make data observable, so problems are caught by monitors before a customer notices them.
• Keep cost in check as data volumes grow, choosing the simplest tool that does the job well.
• Help set the team's data standards and shape the platform as one of its first data engineers.

Skills

5+ years in production data pipelines

Python

Advanced SQL

Data modeling (PostgreSQL)

Data cleaning/normalization

Workflow orchestration

AWS data services

Data validation/testing

Clear communication

Tools

PostgreSQL

pgvector

S3

Airflow

CI/CD for data pipelines

Job description

Foresight is seeking a Senior Data Engineer to build the data foundation behind our platform. You will design and maintain pipelines that ingest project data from customers’ scheduling tools and turn it into clean, well‑modeled data that powers our predictions, analytics and customer reporting.

This hands‑on role focuses on data quality, observability, and governance. You’ll collaborate with ML engineers and full‑stack teams, design for failure, and optimize cost as data volumes grow—shaping