Senior Databricks / Snowflake Data Architect – BestX, VP

State Street; Clifton (NJ); On-site; USD 125,000 - 226,000; Full time

Benefits offered by this job

401K with company match

Health insurance

Paid time off

Job summary

State Street is seeking a Senior Databricks / Snowflake Data Architect to lead the design of a cloud-native data platform for multi-asset TCA, analytics, and reporting. You will drive architecture decisions, prototype new technologies, and build scalable lakehouse solutions with Databricks and Snowflake.

You will collaborate with quantitative researchers and engineering teams to translate business needs into robust data pipelines, governance, and observability while aligning with MiFID II/MiFIR

Qualifications

• Design and implement enterprise-scale data platforms
• Proven experience with Databricks and Snowflake in production
• Strong knowledge of data lakehouse architectures and governance
• Hands-on background with Python, SQL, and modern ELT/ETL frameworks
• Experience with cloud services (AWS) and streaming architectures
• Ability to prototype and benchmark emerging data technologies
• Familiarity with regulatory requirements in financial services (MiFID II/MiFIR)
• Excellent communication and stakeholder management

Responsibilities

• Lead design of multi-asset TCA and analytics platform
• Evaluate technologies and prototype cloud-native data solutions
• Build scalable data lakehouse architectures using Databricks and Snowflake
• Develop high-performance data models, ingestion, and transformation pipelines
• Ensure data governance, lineage, security, and observability
• Collaborate with researchers, product and engineering teams
• Drive CI/CD, IaC, and DataOps practices across the team
• Mentor engineers and guide architecture decisions end-to-end
• Stay aligned with regulatory requirements and best practices

Skills

Databricks

Snowflake

Apache Spark

Python

SQL

ELT/ETL

Data modeling

AWS

CI/CD

Infrastructure as Code

Data governance

Kafka

Spark Streaming

MLOps

MiFID II/MiFIR

Tools

Delta Lake

Unity Catalog

Spark

Snowpark

Data sharing

Kafka

EventBridge

Infrastructure as Code

Job description

We are seeking an experienced Senior Databricks / Snowflake Data Architect to join the BestX engineering team. In this role, you will lead the design and evolution of BestX's next-generation cloud-native data platform, leveraging Databricks, Snowflake, and modern data engineering technologies to support multi-asset Transaction Cost Analysis (TCA), analytics, and reporting capabilities.

The ideal candidate combines strong data architecture expertise with a hands-on engineering mindset and a passion for evaluating emerging technologies. You will help shape the strategic direction of our data platform through rapid prototyping, proof-of-concepts, and scalable production implementations.

Key Responsibilities

Design and develop the architecture for BestX's next-generation multi-asset TCA and analytics platform. Lead technology evaluations, proof-of-concepts, and rapid prototyping initiatives across modern data platforms and cloud-native technologies. Architect and implement scalable data lakehouse solutions using Databricks, Snowflake, and cloud-native services. Design high-performance data models, ingestion frameworks, and transformation pipelines supporting both real-time and batch processing workloads. Establish best practices for data governance, lineage, security, metadata management, and platform observability. Build and optimise analytical platforms capable of processing large-scale market, transaction, and reference datasets. Work closely with quantitative researchers, product teams, and engineering teams to translate business requirements into scalable technical solutions. Define architectural standards and technology roadmaps for data warehousing, lakehouse, streaming, and AI/ML workloads. Drive adoption of modern Agile, DevOps, CI/CD, Infrastructure-as-Code, and DataOps practices. Develop a strong understanding of Transaction Cost Analysis (TCA), best execution workflows, and relevant regulatory requirements including MiFID II and MiFIR. Mentor engineering teams and provide technical leadership across strategic platform initiatives. Take ownership of architecture decisions from concept through production delivery.

Core Skills And Experience

Experience in designing and implementing enterprise-scale data platforms. Strong hands-on experience with Databricks, including Delta Lake, Unity Catalog, Spark, Structured Streaming, and Lakehouse architecture. Strong hands-on experience with Snowflake, including data modelling, performance optimisation, Snowpark, data sharing, and governance capabilities. Expertise in Apache Spark, Python, SQL, and modern ELT/ETL frameworks. Proven ability to rapidly evaluate, prototype, and compare emerging technologies and architectural patterns. Experience designing large-scale analytical platforms supporting high-volume and high-velocity datasets. Strong understanding of data architecture principles including dimensional modelling, Data Vault, lakehouse methodologies, and distributed computing. Experience with cloud platforms, preferably AWS, including services such as S3, EMR, Glue, ECS/EKS, Lambda, and related analytics services. Experience building streaming architectures using technologies such as Kafka, Spark Streaming, EventBridge, or equivalent. Deep understanding of data governance, security, lineage, and regulatory controls in financial services environments. Strong software engineering and DevOps mindset, including CI/CD and Infrastructure-as-Code. Excellent stakeholder management and communication skills, capable of influencing architecture decisions across multiple teams.

Preferred Skills

Experience building analytics platforms for capital markets, including FX, Fixed Income, Equities, or Derivatives. Knowledge of Transaction Cost Analysis (TCA), execution analytics, and market microstructure. Experience with AI/ML platforms, feature stores, and MLOps frameworks. Experience evaluating alternative analytical databases such as DuckDB, ClickHouse, Redshift, BigQuery, KDB+, or Apache Iceberg-based architectures. Experience with data catalogue and governance platforms. Exposure to modern visualisation and BI tools such as Power BI, Tableau, Sigma, or ThoughtSpot. Experience leading cloud migration or platform modernisation initiatives. Experience operating in highly regulated financial services environments.

Salary Range

$125,000 - $226,250 Annual

The range quoted above applies to the role in the primary location specified. If the candidate would ultimately work outside of the primary location above, the applicable range could differ.

Employees are eligible to participate in State Street’s comprehensive benefits program, which includes:

• our retirement savings plan (401K) with company match
• insurance coverage including basic life, medical, dental, vision, long-term disability, and other optional additional coverages
• paid-time off including vacation, sick leave, short term disability, and family care responsibilities
• access to our Employee Assistance Program
• incentive compensation including eligibility for annual performance-based awards (excluding certain sales roles subject to sales incentive plans)
• eligibility for certain tax advantaged savings plans

For a full overview, visit https://hrportal.ehr.com/statestreet/Home.

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at StateStreet.com/careers

Read our CEO Statement

Job Application Disclosure

It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.

Job ID: R-797536