Data Engineer – Tier 1 Data Team – NYC Investment Manager

Mondrian Alpha; New York (NY); On-site; USD 150,000 - 190,000; Full time

Job summary

Mondrian Alpha seeks a Spark Engineer in New York to own data pipelines feeding research and portfolios. You design, build, and optimize large-scale ETL pipelines in Spark and Databricks, turning raw market and alternative data into a trusted asset for analysts.

You will own performance at scale, build Delta Lake architectures, and partner with quants to productionize data needs. This role demands hands-on leadership and deep expertise in Spark, Databricks, and data quality at scale.

Qualifications

• 4+ years hands-on Spark production experience with Databricks.
• Strong PySpark or Scala, plus SQL.
• Delta Lake, job orchestration, and cluster tuning at scale.
• Proven ability to fix fragile pipelines and make them fast and reliable.
• Be the go-to Spark expert in the building, not just a contributor.

Responsibilities

• Design, build, and optimize large-scale ETL pipelines in Spark and Databricks, processing high-volume market and alternative data.
• Own performance at scale: partitioning, caching, cluster sizing, and cost, not just correctness.
• Build the Delta Lake architecture that gives research and quant teams a single, reliable source of truth.
• Partner with quants and researchers to turn data requirements into production-grade pipelines.
• Set and enforce data quality and lineage standards across the Databricks environment.
• Drive the migration of legacy batch jobs onto a modern, scalable Spark architecture.

Skills

PySpark

Scala

SQL

Data quality

Data lineage

Tools

Spark

Databricks

Delta Lake

Structured Streaming

Kafka

AWS

Job description

Every strategy, every research notebook, every risk model at the firm reads from data this engineer's pipelines produce. The role owns the Spark and Databricks infrastructure that turns raw market, reference, and alternative data into something research and portfolio teams can trust without double-checking it. When this pipeline is fast and clean, research moves at the speed of ideas. When it isn't, every team downstream feels it the same day.

What You’ll Do

• Design, build, and optimize large-scale ETL pipelines in Spark and Databricks, processing high-volume market and alternative data
• Own performance at scale: partitioning, caching, cluster sizing, and cost, not just correctness
• Build the Delta Lake architecture that gives research and quant teams a single, reliable source of truth
• Partner with quants and researchers to turn their data requirements into production-grade, reusable pipelines
• Set and enforce data quality and lineage standards across the Databricks environment
• Drive the migration of legacy batch jobs onto a modern, scalable Spark architecture

Must-haves

• 4+ years of hands-on Spark experience in production, with deep fluency in Databricks specifically. This is non-negotiable for the role
• Strong PySpark or Scala, plus the SQL to match
• Real experience with Delta Lake, job orchestration, and cluster performance tuning at scale
• Track record of taking a slow or fragile pipeline and making it fast and boring, in the best sense
• Comfortable being the go-to Spark expert in the building, not just a contributor on someone else's pipeline

Nice-to-haves

• Experience with streaming data (Structured Streaming, Kafka) alongside batch
• Exposure to financial market or tick data specifically
• Familiarity with MLflow or feature stores built on top of Databricks
• AWS or Azure infrastructure experience supporting a Databricks environment

Why This Role

This is the most technically demanding data seat in the building, and it's built that way on purpose. The firm has invested seriously in its Databricks environment and is looking for someone to take full ownership of it, not inherit someone else's half-finished setup. Comp is top of market, reflecting how central this infrastructure is to everything the fund does. For a Spark engineer who wants their pipeline to be the thing every research team depends on and never has to think about, this is that.