Hybrid Senior Databricks Data Engineer – Azure Spark Lead

State Street; Princeton (NJ); Hybrid; USD 120,000 - 203,000; Full time

Job summary

State Street in Princeton is seeking a senior data platform engineer to design, build, and maintain end-to-end data pipelines on Azure, Databricks, and related technologies within an Agile team. You will own ETL processes, data ingestion, and data quality checks, collaborating across DevOps, data science, and software developers while following best practices.

The role requires 10+ years in big data pipelines, Databricks expertise, cloud services, and a hands-on approach in a fast-paced

Qualifications

• Bachelor’s degree in computer science or IT-related subject.
• 10+ years of overall Bigdata data pipeline experience.
• 5+ years of Databricks hands-on experience.
• 5+ years of cloud-based development including Azure Services, Azure Devops, Kubernetes, Docker.
• Experience with Snowflake is a plus.

Responsibilities

• Design, build and test end-to-end data pipeline including ingestion on Azure and Cosmos DB.
• Develop and implement ETL processes for data ingestion, transformation and loading into data lakes.
• Work on Databricks and data warehousing concepts.
• Design and develop high-throughput, configurable frameworks/libraries.
• Collaborate to drive changes and produce POCs.
• Own SDLC activities including design, coding, integration testing, deployment, and documentation.
• Work within an agile project team.
• Follow best practices and coding standards.
• Grow your personal skillset.

Skills

Data pipeline design

ETL development

Databricks

Azure cloud

Cosmos DB

Microservices architecture

Scala

Databricks SQL

Git

Unix shell scripting

DevOps (ADO/Harness)

SQL

Education

Bachelor’s Degree in Computer Science or IT

Tools

Databricks

Delta Lake

Apache Spark

Azure

Kubernetes

Docker

Snowflake

Azure DevOps

Git

Job description

State Street in Princeton is seeking a senior data platform engineer to design, build, and maintain end-to-end data pipelines on Azure, Databricks, and related technologies within an Agile team. You will own ETL processes, data ingestion, and data quality checks, collaborating across DevOps, data science, and software developers while following best practices.

The role requires 10+ years in big data pipelines, Databricks expertise, cloud services, and a hands-on approach in a fast-paced