Senior Cloud Data Lake Engineer & Tech Lead

State Street; Quincy (MA); On-site; USD 90,000 - 157,500; Full time

Benefits offered by this job

401K with company match

Paid time off

Volunteer days

Employee networks

Job summary

State Street is seeking a highly skilled Data Engineer to design, build, and optimize data infrastructure and analytics initiatives. This hands-on technologist will lead data framework development, ensuring data integrity, scalability and security.

In this role, you will work with cloud data lake activities in AWS and Databricks, implement CI/CD pipelines, and guide a global team of engineers. A strong background in Spark, PySpark/Scala, and data modeling is essential for success.

Qualifications

• Bachelor’s/Master's Degree in Computer Science, Engineering or Math.
• 5+ years' experience in professional software design and development.
• Strong analytical skills with ability to absorb and analyze information quickly.
• Proven ability to work with cross-functional processes and timelines.
• Excellent follow-through, attention to detail and time management.
• Adapt and excel at managing multiple tasks in a high-pressure, deadline environment.

Responsibilities

• Collaborate with data scientists, analysts, and product teams to enable analytics and ML use cases.
• Ensure data quality, reliability, security, and governance across data platforms.
• Optimize performance and cost of data workloads in AWS and Databricks.
• Implement CI/CD pipelines and infrastructure as code (e.g., Terraform, CloudFormation).
• Monitor, troubleshoot, and resolve data pipeline and platform issues.
• Document data architecture, pipelines, and operational processes.
• End to end Cloud Data Lake design & development including data ingestion, data modeling and data distribution.
• Build data integrations, hand-offs between on‑prem/cloud‑based systems.
• Design, build, and maintain scalable, reliable data pipelines using AWS services and Databricks.
• Develop and optimize ETL/ELT workflows for batch and near-real-time data processing.
• Implement data solutions using Apache Spark (PySpark/Scala) on Databricks.
• Leverage AWS services such as S3, Glue, Lambda, EMR, Redshift, Athena, and Kinesis.
• Highly motivated, with initiative, prioritization and independent task execution.
• Management oversight of a global team of software development and QA engineers.
• Build strong partnerships with technology & business leaders and peers.
• Work with Product Owner on requirements analysis, planning, and execution.

Skills

Python

Scala/PySpark

SQL/PLSQL

Databricks

AWS

Spark

Terraform

Education

Bachelor’s/Master's Degree in CS/Engineering/Math

Tools

ETL

Cloud platforms

Job description

State Street is seeking a highly skilled Data Engineer to design, build, and optimize data infrastructure and analytics initiatives. This hands-on technologist will lead data framework development, ensuring data integrity, scalability and security.

In this role, you will work with cloud data lake activities in AWS and Databricks, implement CI/CD pipelines, and guide a global team of engineers. A strong background in Spark, PySpark/Scala, and data modeling is essential for success.