Senior Databricks Data Engineer — Real-Time Pipelines

State Street

Quincy (MA)

Hybrid

USD 90,000 - 158,000

Full time

2 days ago

Be an early applicant

Application generator

Stand out for this role — generate a tailored resume and cover letter in about a minute.

Get past ATS filters

Job summary

State Street is seeking a Sr. Databricks Data Engineer to design and develop high-throughput data pipelines on Databricks, processing large data volumes in near real time for State Street Global Technology Services.

You will build streaming, event-driven and batch ingestion pipelines, optimize Databricks jobs, and contribute to the full software development lifecycle. Experience with Azure, API/Event Driven architecture, and cloud-based development is valued.

Qualifications

Bachelor’s degree in computer or IT subject.
7+ years total IT experience.
5+ years Data Engineering handling large datasets.
3+ years Databricks hands-on with SQL & Scala/PySpark.
3+ years cloud-based development on Azure/AWS.

Responsibilities

Design & develop custom high throughput and configurable data pipelines using DataBricks.
Ability to drive change through collaboration, influence and POCs.
Design, build and test end to end data ingestion pipelines (streaming, events, and batch).
Design, develop, and deploy scalable ETL pipelines on cloud-based infrastructure using Databricks SQL & Scala/PySpark.
Optimize and tune Databricks job performance, including query optimization, resource management, and scaling strategies.
Address data issues, environment issues, performing impact analysis, root cause analysis, corrective, adaptive and perfective maintenance.
Responsible for all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation.