Sr. Data Engineer (UP)

OP Recruiting; Chicago (IL); Hybrid; USD 200,000 - 250,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

401(k) matching

Disability insurance

Life/AD&D

HSA

FSA

Job summary

OP Recruiting seeks a Senior Data Engineer to lead the Core Data Platform, transforming external feeds into analytics-grade data assets. You will partner with quantitative researchers and trading desks to build resilient pipelines and governance, ensuring data quality across batch and streaming workloads.

The role emphasizes enterprise-scale architecture, migration experience, and collaboration in a fast-paced financial environment with hybrid Chicago or New York work locations.

Qualifications

• 7+ years of enterprise data platform experience.
• Expert SQL and production-grade Python or Java skills.
• Strong data modeling and architecture judgment.
• Experience with orchestration tools and modern storage formats.
• Proven ability to lead data platform migrations and recoveries.
• Excellent communication in a fast-paced finance/trading environment.

Responsibilities

• Architect, construct, and maintain scalable data pipelines and centralized platform architecture supporting enterprise-wide analytics and quantitative trading workflows.
• Evaluate, ingest, and operationalize external vendor data feeds in collaboration with domain experts.
• Implement robust data governance frameworks, including access control, lineage, schema management, usage tracking, and automated quality metrics.
• Oversee operational stability of ingestion pipelines, managing schema updates, data backfills, dead-letter queue resolution, and rotating on-call support.
• Build automated monitoring and validation tools to preemptively detect and resolve data anomalies across batch and streaming pipelines.
• Drive continuous improvement of internal tooling and self-service capabilities to accelerate data discovery and integration for end users.

Skills

SQL

Python

Java

Data modeling

Communication

Tools

Airflow

Dagster

Prefect

Kubernetes

Argo Workflows

Job description

Senior Data Engineer - Core Data Platform

Location: Chicago, IL or New York, NY (Hybrid)

About The Opportunity

Are you ready to architect high-throughput data infrastructure for a premier global quantitative trading and investment organization? We are seeking a Senior Data Engineer to lead the evolution of our central data ecosystem, transforming complex external market feeds into high-value analytical assets. In this high-impact role, you will partner directly with quantitative researchers, trading desks, and technical leaders to build resilient, scalable data solutions.

Responsibilities

• Architect, construct, and maintain scalable data pipelines and centralized platform architecture supporting enterprise-wide analytics and quantitative trading workflows.
• Evaluate, ingest, and operationalize external vendor data feeds in close collaboration with domain experts and quantitative analysts.
• Implement robust data governance frameworks, including access control, lineage, schema management, usage tracking, and automated quality metrics.
• Oversee operational stability of ingestion pipelines, managing schema updates, data backfills, dead-letter queue resolution, and rotating on-call support.
• Build automated monitoring and validation tools to preemptively detect and resolve data anomalies across batch and streaming pipelines.
• Drive continuous improvement of internal tooling and self-service capabilities to accelerate data discovery and integration for end users.

Requirements (Must-Have)

• 7+ years of dedicated experience building enterprise-grade data platforms and pipeline infrastructure.
• Expert-level mastery of SQL along with production-grade proficiency in either Python or Java.
• Demonstrated expertise in modern data modeling paradigms and architecture trade-offs.
• Hands-on experience orchestrating complex workflows using systems such as Airflow, Dagster, Prefect, or Kubernetes/Argo Workflows.
• Proven experience with modern storage systems and file formats (e.g., Delta Lake, Iceberg, Parquet, S3, ClickHouse, Cassandra, or relational databases).
• Track record of executing non-trivial data platform migrations and managing operational recovery processes (batch reprocessing, backfills, queue monitoring).
• Strong communication skills and the ability to thrive in a collaborative, fast-paced trading/financial environment.

Preferred Qualifications (Nice-to-Have)

• Familiarity with distributed compute and streaming frameworks (e.g., Flink, Spark, Polars, Dask, Kafka).
• Experience handling binary serialization standards such as Avro, Arrow, Protobuf/gRPC, or Thrift.
• Background in financial markets, market data ingestion, or high-throughput analytics platforms.

Compensation & Benefits

• Base Salary Range: $200,000 – $250,000 per year (commensurate with experience and skill set).
• Bonus: Eligible for an annual discretionary performance bonus.
• Benefits: Comprehensive package including medical, dental, and vision insurance; matching 401(k); short/long-term disability; life and AD&D coverage; health savings accounts (HSA); and flexible spending accounts (FSA).

Accepting Candidates