Senior Databricks Data Engineer - Real-Time Pipelines

Statestreet; Toronto; Hybrid; CAD 84,000 - 141,000; Full time

Job summary

State Street is seeking a Data Engineer to design and implement high-throughput data pipelines using DataBricks for near real-time processing. You will build end-to-end ingestion pipelines (streaming, events, batch) and deploy scalable ETL on cloud infrastructure with Databricks SQL and PySpark/Scala.

The role requires strong troubleshooting, collaboration across teams, and experience with Azure/AWS services. Onsite 4 days per week, one day remote, within a hybrid work model.

Qualifications

• Bachelor’s degree in computer or IT related subject.
• 7+ years of IT experience.
• 5+ years of Data Engineering handling large datasets.
• 3+ years of Databricks hands-on experience (SQL & Scala/PySpark).
• 3+ years of cloud-based development using Azure/AWS.

Responsibilities

• Design and develop high throughput data pipelines using DataBricks.
• Build end-to-end data ingestion pipelines (streaming, events, batch).
• Develop scalable ETL pipelines on cloud infrastructure using Databricks SQL and Scala/PySpark.
• Optimize Databricks job performance including query optimization and scaling strategies.
• Handle data issues, root cause analysis, and maintenance across environments.
• Participate in all SDLC phases from design to deployment and documentation.

Skills

Databricks SQL

Scala/PySpark

Azure

AWS

Data Lakehouse

API/Event Driven Architecture

DevOps pipelines

Communication

Team collaboration

Education

Bachelor’s Degree in Computer or IT

7+ years IT experience

5+ years Data Engineering

3+ years Databricks experience (SQL & Scala/PySpark)

3+ years Cloud development Azure/AWS

Tools

Databricks

Azure

AWS

Job description

State Street is seeking a Data Engineer to design and implement high-throughput data pipelines using DataBricks for near real-time processing. You will build end-to-end ingestion pipelines (streaming, events, batch) and deploy scalable ETL on cloud infrastructure with Databricks SQL and PySpark/Scala.

The role requires strong troubleshooting, collaboration across teams, and experience with Azure/AWS services. Onsite 4 days per week, one day remote, within a hybrid work model.