Lead Cloud Data Lake Engineer: Spark, AWS & Databricks

STATE STREET CORPORATION; Quincy (MA); On-site; USD 90,000 - 157,500; Full time

Job summary

State Street Corporation is seeking a highly skilled Data Engineer to design, build, and optimize our data infrastructure and analytics initiatives. As a hands-on technology lead, you will contribute to the design, development, enhancement & maintenance of data frameworks, ensuring data integrity, scalability and security.

You will collaborate with data scientists, analysts, and product teams to enable analytics and ML use cases, build scalable data pipelines on AWS and Databricks, and implement

Qualifications

• 5+ years' experience in professional software design and development.
• Strong analytical skills with ability to learn quickly and work independently.
• Experience with cloud platforms, AWS and Databricks.
• Hands-on experience with data lakes, data modeling, and data governance.

Responsibilities

• Collaborate with data scientists, analysts, and product teams to enable analytics and ML use cases.
• Build scalable, reliable data pipelines using AWS services and Databricks.
• Design, develop, and maintain end-to-end Cloud Data Lake including ingestion, modeling and distribution.
• Implement CI/CD pipelines and infrastructure as code (Terraform, CloudFormation).
• Monitor, troubleshoot, and resolve data pipeline and platform issues.

Skills

Python

Scala/PySpark

PL/SQL

Spark

Databricks

AWS

Data modeling

ETL/ELT

Education

Bachelor's/Master's in CS/Engineering/Math

Tools

Git

CI/CD

Job description

State Street Corporation is seeking a highly skilled Data Engineer to design, build, and optimize our data infrastructure and analytics initiatives. As a hands-on technology lead, you will contribute to the design, development, enhancement & maintenance of data frameworks, ensuring data integrity, scalability and security.

You will collaborate with data scientists, analysts, and product teams to enable analytics and ML use cases, build scalable data pipelines on AWS and Databricks, and implement