Data Platform Engineer - Spark/Databricks for Investment

Mondrian Alpha; New York (NY); On-site; USD 140,000 - 200,000; Full time

Job summary

Mondrian Alpha in New York City is seeking a data engineer to own scalable batch and streaming pipelines built on Databricks and Spark. You will work closely with researchers, risk managers, and portfolio teams to translate data needs into production-grade datasets and models.

You’ll be responsible for data quality, monitoring, and cost/performance optimization, while shaping how the platform evolves to support investment decisions.

Qualifications

• 2–10 years of data engineering experience.
• Strong Python, SQL, and Spark experience.
• Excellent ability to communicate with non engineers and stakeholders.

Responsibilities

• Design, build, and maintain large scale batch and streaming pipelines in Spark on Databricks.
• Own the full lifecycle of datasets from vendor onboarding to models and tables consumed by researchers.
• Collaborate with researchers, portfolio managers, and risk to translate data questions into concrete engineering work.
• Build monitoring, testing, and data quality controls to catch problems early.
• Improve performance and cost of Spark workloads and evolve the platform accordingly.
• Document and explain work to technical and non-technical audiences.

Skills

Python

SQL

Stakeholder communication

Cloud data platforms

Tools

Databricks

Spark

Airflow / orchestration tooling

Delta Lake / lakehouse

Job description

Mondrian Alpha in New York City is seeking a data engineer to own scalable batch and streaming pipelines built on Databricks and Spark. You will work closely with researchers, risk managers, and portfolio teams to translate data needs into production-grade datasets and models.

You’ll be responsible for data quality, monitoring, and cost/performance optimization, while shaping how the platform evolves to support investment decisions.