Senior Spark Engineer

Rose International; Jersey City (NJ); Sur place; USD 180 000 - 230 000; Plein temps

Résumé du poste

Rose International seeks a Lead Engineer for Big Data Risk Aggregation to own the core platform that transforms raw calculation output into final, reportable risk numbers. You will build and run industrial-scale data pipelines that power risk reporting and analytics for global regulators.

You will lead a team of data engineers, developing a metadata-driven aggregation platform using Spark, and delivering fast, auditable results for multi-billion-dollar capital decisions.

Qualifications

• 10+ years in hands-on, large-scale data engineering in capital markets or risk tech.
• Expert-level Spark with performance tuning and partitioning.
• Deep knowledge of VaR, CCAR, Stress Testing, FRTB.
• Strong Java and/or Python, expert SQL.
• Experience building semantic data models bridging raw data to BI/AI.

Responsabilités

• Architect and build high-performance, metadata-driven aggregation platform with Spark.
• Design algorithms to reduce high-dimensional risk calculation complexity.
• Build a semantic data layer for BI and AI queries.
• Optimize Spark partitioning and minimize network shuffles for performance.
• Create a self-service platform for quants to define new aggregations.
• Develop APIs and datasets for notebooks to enable analysis.
• Load metrics into Pinot/Druid for analytics.

Connaissances

Apache Spark expert

Java/Python programming

SQL expert

Risk aggregation

Data modeling / semantic layer

High‑performance data pipelines

Problem solving at scale

Formation

Quant/Technical degree

Outils

Apache Pinot

Apache Druid

Description du poste

Required Education

• A degree in a quantitative or technical field such as Computer Science, Financial Mathematics, or Financial Engineering.

Required Qualifications/Skills/Experience

• 10+ years of professional experience in hands‑on, large‑scale data engineering, with a significant portion within the Capital Markets or Risk technology space.
• Expert‑level, hands‑on experience with Apache Spark, including deep knowledge of its internal architecture, performance tuning, shuffle optimization, and partitioning strategies.
• Deep domain knowledge of risk aggregation methodologies for VaR, CCAR, Stress Testing, and FRTB is essential.
• Strong programming skills in Java and/or Python, and expert‑level SQL.
• Experience designing and building semantic layers or data models that bridge the gap between complex physical data and business intelligence/AI applications.
• A knack for solving complex data reconciliation and performance problems at scale.
• You are a rare blend of a deep computer scientist and a savvy financial engineer, with a passion for solving problems from first principles.

Preferred Qualifications/Skills/Experience

• Proven experience with high‑performance OLAP databases such as Apache Pinot or Apache Druid.
• An obsession with performance metrics, from CPU cycles and memory footprints to network I/O.

Overview

Are you a computer scientist who loves finance? Do you want to solve high‑complexity algorithmic challenges on one of the world’s largest financial datasets? This isn’t about maintaining legacy systems; it’s a green‑field opportunity to build a brand‑new, best‑in‑class aggregation engine from first principles, using the most modern tools available. Client is seeking an expert Lead Engineer for Big Data Risk Aggregation to own the core platform that transforms raw calculation output into final, reportable risk numbers. You will be responsible for building and running the industrial‑scale data pipelines that sit at the heart of our risk reporting and analytics capabilities.

As the Lead Engineer for Big Data Risk Aggregation, you are the master of large‑scale data transformation. You will take the immense output from our pricing and calculation engines—billions of P&L vectors across thousands of scenarios—and apply the complex aggregation logic required for our most important risk measures. Your primary mission is to build a platform that is not only incredibly fast and scalable but also robust, auditable, and intelligent. The code you write and the algorithms you design will directly influence multi‑billion‑dollar capital decisions and be reported to global regulators. You will lead a team of expert data engineers, setting the standard for big data processing and delivering the results that drive strategic capital and risk management decisions.

Job Duties

• Architect and build a high‑performance, metadata‑driven aggregation platform using Apache Spark.
• Design and implement novel algorithms to reduce the computational complexity of high‑dimensional risk aggregations.
• Design and build a rich semantic layer over the aggregated data, creating a business‑friendly data model that can power both traditional BI and next‑generation AI‑driven queries.
• Solve deep computer science challenges within Spark, choosing the right underlying mechanisms, designing optimal data partitioning strategies, and minimizing network shuffling to ensure extreme performance.
• Build a self‑service platform that empowers quants and risk managers to define new aggregations without writing new code.
• Pioneer innovative ways of making data available to users by building high‑performance APIs for UIs, and providing curated datasets for Python Notebooks that enable deep forensic analysis and calculation replication.
• Load the final, aggregated risk metrics into high‑performance OLAP stores (Apache Pinot, Apache Druid) to serve the analytics and UI layers.
• Only those lawfully authorized to work in the designated country associated with the position will be considered.
• Please note that all Position start dates and duration are estimates and may be reduced or lengthened based upon a client’s business needs and requirements.

About Rose

• Founded in 1993
• Office Locations Across the U.S.
• 150+ Clients: Corporations and Government Agencies
• Employee Oriented Company
• Challenging Assignments Across the U.S.
• Continuous Professional Development

EMPLOYEE COMMENTS

• We want you to work with us, but don’t take our word for it. Take a look at this sampling of employee comments. They speak for themselves.