Senior Databricks Data Engineer Real-Time ETL & Lakehouse

State Street; Quincy (CA); On-site; USD 90,000 - 158,000; Full time

Job summary

State Street is seeking a Sr. Databricks Data Engineer to design and deploy high-throughput data pipelines on Databricks. You will build end-to-end data ingestion (streaming, events, batch) and drive scalable ETL on Azure/AWS cloud platforms, while optimizing performance and leading SDLC activities.

Ideal candidates have 7+ years in IT and 3+ years with Databricks SQL & Scala/PySpark, plus strong collaboration and communication skills to work with global teams.

Qualifications

• Bachelor's Degree in a computer or IT related subject.
• 7+ years of overall IT experience.
• 5+ years of Data Engineering experience handling large datasets.
• 3+ years of Databricks hands-on experience using SQL & Scala/PySpark.
• 3+ years of experience on cloud-based development using Azure/AWS.

Responsibilities

• Design, develop, and deploy high throughput data pipelines using DataBricks.
• Build end-to-end data ingestion pipelines (streaming, events, batch) and data curation.
• Optimize Databricks job performance, including query optimization and scaling.
• Address data issues, environment issues, performing impact analysis, root cause analysis.
• Responsible for all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation.

Skills

Databricks SQL

Scala/PySpark

Data pipelines

Azure services

DevOps troubleshooting

Problem solving

Communication

Education

Bachelor's Degree in Computer/IT

Tools

Azure

AWS

Databricks

Job description

State Street is seeking a Sr. Databricks Data Engineer to design and deploy high-throughput data pipelines on Databricks. You will build end-to-end data ingestion (streaming, events, batch) and drive scalable ETL on Azure/AWS cloud platforms, while optimizing performance and leading SDLC activities.

Ideal candidates have 7+ years in IT and 3+ years with Databricks SQL & Scala/PySpark, plus strong collaboration and communication skills to work with global teams.