Senior Databricks Data Engineer - Real-Time Data Pipelines

State Street; Clifton (NJ); Hybrid; USD 90,000 - 158,000; Full time

Benefits offered by this job

401K with company match

Health, life, medical, dental, vision,

Long-term disability

Paid time off

Employee Assistance Program

Incentive compensation

Tax-advantaged savings plans

Job summary

State Street is seeking a Sr. Databricks Data Engineer to build and optimize data pipelines on Databricks, enabling near real-time data processing for State Street Global Technology Services.

You will design, develop and deploy scalable ETL pipelines on cloud-based infrastructure, leveraging Databricks SQL, Scala and PySpark to deliver high-throughput data solutions. Strong collaboration and DevOps-oriented mindset are essential.

Qualifications

• Bachelor’s degree in computer/IT field required.
• 7+ years IT experience; 5+ years data engineering.
• 3+ years in Databricks with SQL & Scala/PySpark.
• 3+ years cloud development with Azure/AWS.

Responsibilities

• Design and develop high-throughput data pipelines on Databricks.
• Drive change through collaboration and POCs.
• Build end-to-end data ingestion (streaming, events, batch).
• Develop ETL pipelines on cloud infrastructure using Databricks SQL and Scala/PySpark.
• Tune performance, manage resources, and scale pipelines.
• Perform root-cause analysis and maintenance across SDLC.

Skills

Databricks SQL

Scala/PySpark

Azure

AWS

Data pipelines

Data Lakehouse

DevOps tooling

API/Event Driven

Communication

Education

Bachelor’s Degree in Computer/IT

Tools

Databricks

Azure

AWS

Job description

State Street is seeking a Sr. Databricks Data Engineer to build and optimize data pipelines on Databricks, enabling near real-time data processing for State Street Global Technology Services.

You will design, develop and deploy scalable ETL pipelines on cloud-based infrastructure, leveraging Databricks SQL, Scala and PySpark to deliver high-throughput data solutions. Strong collaboration and DevOps-oriented mindset are essential.