Data Platform - Capability Engineer - Assistant Vice President

State Street Corporation; Bengaluru; On-site; INR 3,800,000 - 6,500,000; Full time

Job summary

State Street Corporation in Bengaluru seeks a Senior Data Engineering Lead to design, build, and operate a modern data platform supporting investment management. You will lead data engineering efforts across batch and streaming pipelines, ensure governance and security, and partner with product and analytics teams to deliver scalable, AI-ready data assets.

The role emphasizes hands-on coding (Python/SQL), leadership of a small team, and delivering reliable, cost-efficient platform solutions on

Qualifications

• 14+ years in software/data engineering; BS/MS in CS/Engineering (or equivalent experience).
• Strong hands-on coding in Python and SQL (Java/Scala a plus).
• Proven delivery of lakehouse/data platform solutions using Databricks and/or Snowflake on AWS (GCP/Azure acceptable).
• Strong Spark (batch/streaming) development and performance/cost tuning.
• Experience with Apache Iceberg and table/catalog governance concepts (e.g., Unity Catalog).
• Production engineering practices: CI/CD, IaC (Terraform/CloudFormation), automated testing, security basics, and observability.
• Ownership mindset for operations: on-call/incident response, RCA, backfills, and reliability improvements; strong communication with global teams.

Responsibilities

• Design, build, and operate modern data platform capabilities across lakehouse, batch, streaming, metadata/catalog, observability, security, and governed consumption layers.
• Develop scalable batch and streaming data pipelines using Apache Spark, Snowflake, and/or Databricks, applying strong engineering practices for performance, reliability, reusability, and cost efficiency.
• Implement and optimize Apache Iceberg tables and open data formats, including partitioning, compaction, metadata management, and cross‑engine performance tuning.
• Apply disciplined software and data engineering practices across CI/CD, Infrastructure as Code, automated testing, schema/version management, data quality controls, and end‑to‑end observability.
• Partner with Product, Analytics, ML, and business SMEs to define data semantics, data product contracts, schemas, SLAs, documentation, access controls, and governed serving models.
• Engineer data governance capabilities across lineage, observability, permissions, entitlement enforcement, sensitive data classification, masking/tokenization, auditability, and policy‑based controls.
• Create AI‑ready data assets by enforcing standards for completeness, accuracy, validity, consistency, timeliness, uniqueness, lineage, provenance, metadata quality, and semantic consistency.
• Optimize platform performance and cost through workload tuning, partitioning/clustering, caching, lifecycle management, retention/purge strategies, and query optimization.
• Own operational stability by troubleshooting production issues, supporting backfills, participating in incident response, driving root‑cause fixes, and improving platform reliability.
• Provide technical leadership for a small team or pod, including planning, mentoring, delivery oversight, engineering standards, and operational excellence.

Skills

Python

SQL

Spark

Cloud computing

Data pipeline engineering

Data platform architecture

Software engineering

On-call incident response

Cross-team communication

Education

BS/MS in CS/Engineering

Tools

Databricks

Snowflake

Apache Iceberg

Terraform

CloudFormation

CI/CD pipelines

Kubernetes

Unity Catalog

Job description

Role Summary

State Street Investment Management’s Data, Analytics & AI Services (DAAIS) team is seeking a Senior Data Engineering Lead (Assistant Vice President –2) for our Bangalore, India office. This role is accountable for the design, build, and operate of modern data platform and software capabilities that enable investment management business. The successful candidate will bring deep hands‑on expertise in large-scale data platform architecture, distributed computing, data pipeline engineering, and modern cloud architecture. This role will execute on the broader platform roadmap in partnership with senior leadership and global stakeholders and may lead a small team/pod to deliver reliable, scalable platform outcomes.

Key Responsibilities

• Design, build, and operate modern data platform capabilities across lakehouse, batch, streaming, metadata/catalog, observability, security, and governed consumption layers.
• Develop scalable batch and streaming data pipelines using Apache Spark, Snowflake, and/or Databricks, applying strong engineering practices for performance, reliability, reusability, and cost efficiency.
• Implement and optimize Apache Iceberg tables and open data formats, including partitioning, compaction, metadata management, and cross‑engine performance tuning.
• Apply disciplined software and data engineering practices across CI/CD, Infrastructure as Code, automated testing, schema/version management, data quality controls, and end‑to‑end observability.
• Partner with Product, Analytics, ML, and business SMEs to define data semantics, data product contracts, schemas, SLAs, documentation, access controls, and governed serving models.
• Engineer data governance capabilities across lineage, observability, permissions, entitlement enforcement, sensitive data classification, masking/tokenization, auditability, and policy‑based controls.
• Create AI‑ready data assets by enforcing standards for completeness, accuracy, validity, consistency, timeliness, uniqueness, lineage, provenance, metadata quality, and semantic consistency.
• Optimize platform performance and cost through workload tuning, partitioning/clustering, caching, lifecycle management, retention/purge strategies, and query optimization.
• Own operational stability by troubleshooting production issues, supporting backfills, participating in incident response, driving root‑cause fixes, and improving platform reliability.
• Provide technical leadership for a small team or pod, including planning, mentoring, delivery oversight, engineering standards, and operational excellence.

Required Qualifications

• 14+ years in software/data engineering; BS/MS in CS/Engineering (or equivalent experience).
• Strong hands‑on coding in Python and SQL (Java/Scala a plus).
• Proven delivery of lakehouse/data platform solutions using Databricks and/or Snowflake on AWS (GCP or Azure acceptable).
• Strong Spark (batch/streaming) development and performance/cost tuning.
• Experience with Apache Iceberg and table/catalog governance concepts (e.g., "Unity Catalog" or equivalent).
• Production engineering practices: CI/CD, IaC (Terraform/CloudFormation), automated testing, security basics, and observability.
• Ownership mindset for operations: on‑call/incident response, RCA, backfills, and reliability improvements; strong communication with global teams.

Preferred Qualifications

• Financial services experience (asset management preferred).
• Experience across structured + alternative/unstructured data, plus governance tooling (Snowflake Horizon/Unity Catalog/Collibra/Alation or similar).
• Use of AI‑assisted coding tools (e.g., "GitHub Copilot", "Claude Code") while maintaining code quality.

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success. We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work‑life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

Equal Opportunity Employer

We consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.