Senior Databricks Engineer Real-Time Pipelines in Azure

STATE STREET CORPORATION; Quincy (MA); On-site; USD 90,000 - 158,000; Full time

Job summary

State Street Global Technology Services is seeking a Sr. Databricks Data Engineer to develop high-throughput data pipelines using Databricks SQL, Scala and PySpark. The role includes end-to-end design, testing and deployment of data ingestion pipelines (streaming, events and batch) on cloud infrastructure.

The candidate will optimize performance, manage resources, and troubleshoot DevOps and Azure services while collaborating across teams in a fast-paced financial technology environment.

Qualifications

• Bachelor’s Degree in computer science or a related field.

Responsibilities

• Design, build and test end-to-end data ingestion pipelines (streaming, events and batch).
• Design and deploy scalable ETL pipelines on cloud infrastructure using Databricks SQL & Scala/PySpark.
• Tune Databricks job performance, including query optimization, resource management and scaling strategies.
• Address data issues, perform impact/root-cause analysis, and perform maintenance and improvements.

Skills

Databricks SQL

Scala

PySpark

Data pipeline design

DevOps troubleshooting

Azure/AWS cloud

API/Event driven architecture

Communication

Education

Bachelor’s Degree in computer science or related field

Tools

Databricks

Azure

AWS

SQL

Job description

State Street Global Technology Services is seeking a Sr. Databricks Data Engineer to develop high-throughput data pipelines using Databricks SQL, Scala and PySpark. The role includes end-to-end design, testing and deployment of data ingestion pipelines (streaming, events and batch) on cloud infrastructure.

The candidate will optimize performance, manage resources, and troubleshoot DevOps and Azure services while collaborating across teams in a fast-paced financial technology environment.