Data Engineer - Assistant Manager

State Street; Bengaluru; On-site; INR 1,800,000 - 2,500,000; Full time

Job summary

State Street is seeking a Senior Associate, Data Engineer for our Bangalore, India office. You will develop and maintain scalable data pipelines using PySpark, Spark SQL, and Databricks in a modern data platform environment.

You will work with senior engineers and business stakeholders on design, development, testing and production support, focusing on data quality, governance, and robust ingestion patterns across cloud environments.

Qualifications

• Bachelor's degree or equivalent in CS/Engineering/IS or related field.
• Hands-on experience with Databricks notebooks, workflows, clusters and repos.
• Strong understanding of Apache Spark transformations, joins, partitioning, caching and performance tuning.
• Proficiency in Python (PySpark) and SQL.
• Experience building production data pipelines in cloud environments.
• Familiarity with version control, testing, CI/CD and data governance concepts.

Responsibilities

• Develop and maintain batch and streaming data pipelines using PySpark, Spark SQL, and Databricks.
• Build and support data processing solutions following Bronze/Silver/Gold architecture patterns.
• Implement and maintain data ingestion pipelines from relational DBs, APIs, event streams, and cloud storage.
• Develop incremental processing, CDC, and data transformation workflows.
• Write high-quality, scalable code following engineering standards and best practices.
• Participate in code reviews, testing, deployment and production support activities.
• Monitor data pipeline performance and assist in troubleshooting and root cause analysis.
• Support data quality, metadata and governance requirements.

Skills

Python (PySpark)

SQL

Databricks

Apache Spark

Data Pipelines

Cloud (Azure/AWS)

Airflow

CDC / Change Data Capture

Data Lake / Lakehouse

CI/CD / Code quality

Team collaboration

Education

Bachelor's degree in Computer Science/Engineering/IS or related field

Tools

Databricks

Airflow

Azure Data Factory

Kafka

Kinesis

AWS S3 / ADLS

Job description

Role Summary

State Street Investment Management's Data, Analytics & AI Services (DAAIS) team is seeking a

Senior Associate, Data Engineer

Data, Analytics & AI Services (DAAIS)

State Street Investment Management's Data, Analytics & AI Services (DAAIS) team is seeking a Senior Associate, Data Engineer for our Bangalore, India office. This role is responsible for developing, enhancing, and supporting modern data platform capabilities that enable the investment management business. The successful candidate will have hands‑on experience in data engineering, distributed data processing, cloud‑based data platforms, and software development practices.

Working closely with senior engineers, architects, and business stakeholders, the individual will contribute to the design, development, testing, and operational support of scalable data pipelines and data products across the enterprise data ecosystem.

Key Responsibilities

• Develop and maintain batch and streaming data pipelines using PySpark, Spark SQL, and Databricks.
• Build and support data processing solutions following Bronze/Silver/Gold (multi-hop) architecture patterns.
• Implement and maintain data ingestion pipelines from multiple sources including relational databases, APIs, event streams, and cloud object storage.
• Develop incremental processing, Change Data Capture (CDC), and data transformation workflows.
• Write high-quality, scalable, and maintainable code adhering to engineering standards and best practices.
• Participate in code reviews, testing, deployment, and production support activities.
• Monitor data pipeline performance and assist in troubleshooting, root cause analysis, and issue resolution.
• Support implementation of data quality, metadata, and governance requirements.

Required Qualifications

• 3-7 years of experience in software engineering, data engineering, or related technical roles.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field (or equivalent experience).
• Hands‑on experience with Databricks, including notebooks, workflows, clusters, and repositories.
• Good understanding of Apache Spark concepts including transformations, joins, partitioning, caching, and performance optimization.
• Proficiency in Python (PySpark) and SQL.
• Experience building and supporting production data pipelines in cloud environments.
• Familiarity with software engineering practices including version control, testing, CI/CD, and code management.
• Working knowledge of data lake and lakehouse concepts.
• Experience with orchestration tools such as Databricks Workflows, Airflow, Azure Data Factory, or similar technologies.
• Familiarity with batch, CDC, and streaming ingestion patterns.
• Experience working with Azure and/or AWS cloud platforms.
• Understanding of cloud storage technologies such as AWS S3 or Azure Data Lake Storage.
• Strong analytical, troubleshooting, and problem‑solving skills.
• Ability to work effectively in a collaborative, global team environment.

Preferred Qualifications

• Experience in financial services or asset management environments.
• Exposure to Apache Iceberg, Delta Lake, Unity Catalog, or similar governance technologies.
• Experience with streaming technologies including Kafka, Event Hubs, Kinesis, or Structured Streaming.
• Exposure to data quality, metadata management, and data governance concepts.
• Experience supporting machine learning or analytics use cases through data engineering solutions.
• Knowledge of monitoring, observability, and operational support practices for data platforms.

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work‑life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at StateStreet.com/careers

Read our CEO Statement