Senior Databricks Data Engineer — Real-Time Data Pipelines

STATE STREET CORPORATION; Quincy (MA); On-site; USD 90,000 - 158,000; Full time

Benefits offered by this job

401K with company match

Health benefits

Paid time off

Job summary

State Street Corporation is seeking a Sr. Databricks Data Engineer to design and develop high-throughput data pipelines on Databricks, processing large data volumes in near real time.

You will build end-to-end ingestion pipelines (streaming, events, batch) and curate data on cloud platforms. You will optimize Databricks jobs, address data and environment issues, and contribute to all phases of the software development lifecycle including testing and documentation.

Qualifications

• Bachelor’s degree in computer or IT subjects.
• 7+ years of IT experience.
• 5+ years of Data Engineering handling large datasets.
• 3+ years Databricks hands-on SQL & Scala/PySpark.
• 3+ years cloud development with Azure/AWS.

Responsibilities

• Design and develop high-throughput data pipelines with Databricks.
• Build end-to-end ingestion pipelines (streaming, events, batch) and data curation.
• Tune performance, manage resources, and scale Databricks jobs.
• Collaborate to drive changes, demonstrate POCs.
• Participate in SDLC including design, coding, testing, deployment and documentation.

Skills

Strong collaboration

Excellent communication

Problem-solving

Critical thinking

Team player

Education

Bachelor’s Degree in Computer/IT

Tools

Databricks

Databricks SQL

Scala/PySpark

Azure

AWS

Data Lakehouse

API/Event Driven

DevOps pipelines

Job description

State Street Corporation is seeking a Sr. Databricks Data Engineer to design and develop high-throughput data pipelines on Databricks, processing large data volumes in near real time.

You will build end-to-end ingestion pipelines (streaming, events, batch) and curate data on cloud platforms. You will optimize Databricks jobs, address data and environment issues, and contribute to all phases of the software development lifecycle including testing and documentation.