Data Engineer - Cloud Lakehouse & AI Workflows

State Street; Boston (MA); On-site; USD 70,000 - 119,000; Full time

Job summary

State Street is seeking a Data Engineer - Officer to contribute to architecture, engineering execution, modernization strategy, and technology roadmaps for the Investment Management data platform. You will design and implement large-scale data pipelines using Python, PySpark, and SQL, and build robust ingestion and transformation workflows with Databricks and Snowflake on AWS.

Collaboration across cross‑functional teams and continued adoption of AI-assisted coding will be expected.

Qualifications

• Master’s degree in Computer Science or a related engineering discipline.
• 6-10 years of technology experience with Data Engineering focus.
• Strong hands‑on in Python, Spark, SQL, Databricks and Snowflake, with the ability to design, debug, and optimize distributed data processing.
• Experience building Lakehouse platforms using Snowflake and Databricks on AWS (Azure acceptable).
• Proficiency in data engineering and automation.
• 5+ years of hands‑on experience building data pipelines on cloud platforms.
• Some experience with modern Data Platform Architecture, Lakehouse, Data Products, and cloud‑native analytics ecosystems.
• Good understanding of Data Warehousing, Data Lakes, and Data Quality concepts.
• Experience with unstructured data sources and processing patterns.
• Exposure with GIT, CI/CD Process for Deployment and Releases.
• Strong SDLC, observability, reliability engineering, and incident management practices.
• Familiarity with AI‑assisted development tools (GitHub Copilot, Codex, Anthropic tools).

Responsibilities

• Design, develop, test, and maintain data pipelines using Python, PySpark, and SQL.
• Design, build, and manage robust workflows and data ingestion, transformation using Databricks and Snowflake
• Hands‑on experience with Databricks, Snowflake on AWS (Azure acceptable)
• Design and maintain CI/CD pipelines using Jenkins, Artifactory, GitHub.
• Develop containerized applications for APIs, event based, streaming and batch using Docker.
• Work in Agile project management practices
• Analyze, design, develop, test, and debug with minimal supervision
• Participate in cross‑team group activities to complete assignments.
• Work with cross‑functional teams on core platform capabilities
• Develop Iceberg, unity catalog based tables for consistent access across engines
• Translate business requirements into technical solutions with product owners and analytics teams
• Apply AI code generation tools to accelerate development, improve quality, and productivity
• Excellent verbal and written communication with stakeholders

Skills

Python

Spark

SQL

Databricks

Snowflake

CI/CD

GitHub

Containerization

Education

Master’s degree in Computer Science or related engineering

Tools

Databricks

Snowflake

AWS

Azure

Docker

Jenkins

Artifactory

GitHub

Job description

State Street is seeking a Data Engineer - Officer to contribute to architecture, engineering execution, modernization strategy, and technology roadmaps for the Investment Management data platform. You will design and implement large-scale data pipelines using Python, PySpark, and SQL, and build robust ingestion and transformation workflows with Databricks and Snowflake on AWS.

Collaboration across cross‑functional teams and continued adoption of AI-assisted coding will be expected.