Senior Databricks Data Engineer — Hybrid Real-Time Pipelines

State Street; Princeton (NJ); Hybrid; USD 90,000 - 158,000; Full time

Benefits offered by this job

401K match

Medical coverage

Paid time off

Flexible work hours

Job summary

State Street is seeking a Sr. Databricks Data Engineer to design and implement high-throughput data pipelines in near real time for State Street Global Technology Services.

You will build streaming, event-driven, and batch pipelines using Databricks SQL and Scala/PySpark on cloud infrastructure, optimize job performance, and participate in all SDLC phases with a focus on reliability and scalability.

Qualifications

• Bachelor’s Degree in a computer or IT field.
• 7+ years of overall IT experience.
• 5+ years in Data Engineering with large datasets.
• 3+ years Databricks hands-on with SQL & Scala/PySpark.
• 3+ years cloud-based development using Azure/AWS.

Responsibilities

• Design and develop high-throughput data pipelines using DataBricks.
• Drive change through collaboration and POCs.
• Build end-to-end data ingestion pipelines (streaming, events, batch).
• Develop ETL pipelines on cloud infra with Databricks SQL, Scala/PySpark.
• Tune performance, manage resources, and scale Databricks jobs.
• Handle data issues, root cause analysis, and maintenance tasks.
• Own SDLC activities: design, coding, testing, deployment, docs.

Skills

Databricks SQL

Scala

PySpark

Data pipelines

Azure services

DevOps troubleshooting

Communication

Education

Bachelor’s Degree in Computer/IT

Tools

Databricks

Azure

AWS

SQL

Scala

PySpark

Job description

State Street is seeking a Sr. Databricks Data Engineer to design and implement high-throughput data pipelines in near real time for State Street Global Technology Services.

You will build streaming, event-driven, and batch pipelines using Databricks SQL and Scala/PySpark on cloud infrastructure, optimize job performance, and participate in all SDLC phases with a focus on reliability and scalability.