Delivery Lead – Data Engineering & Databricks (Architecture Focus) - Assistant Vice President

STATE STREET CORPORATION; Hyderabad; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

STATE STREET CORPORATION is seeking an Delivery Lead to drive end-to-end delivery of data engineering platforms with a core focus on Databricks and cloud architecture. You will own solution design, governance, and cross-functional leadership across global teams to ensure scalable, secure data solutions that align with business outcomes.

Ideal candidates have 15+ years in technology delivery, strong Python/Scala and SQL, and hands-on Databricks experience on AWS/Azure.

Qualifications

• 15+ years of technology delivery with leadership exposure.
• Extensive Data Engineering and Big Data experience (Hadoop, Spark, Databricks).
• Strong Python/Scala and SQL programming.
• Cloud platform experience (AWS or Azure).
• CI/CD, DevOps, and data pipeline automation.

Responsibilities

• Own end-to-end delivery of data engineering initiatives with governance.
• Define scalable data architecture using Databricks and cloud platforms.
• Lead design discussions translating business requirements into architecture.
• Establish Agile release planning and risk management.
• Provide leadership to globally distributed teams and report progress.

Skills

Databricks

Spark (Python/Scala)

Python/SQL

Leadership

CI/CD

Education

Bachelor’s or Master’s in CS/Engineering

Tools

Databricks

AWS

Azure

Hadoop

Docker

Kubernetes

Job description

Role Summary & Role Description

The Delivery Lead will drive end-to-end delivery of data engineering platforms with a strong focus on Databricks and modern cloud-based architectures. This role combines delivery leadership with deep technical ownership of solution architecture, ensuring scalable, secure, and high-performing data solutions aligned to business outcomes. The individual will lead cross-functional global teams and act as a bridge between business stakeholders, architects, and engineering teams.

Key Responsibilities

• Own and drive end-to-end delivery of data engineering and analytics initiatives with strong governance and execution discipline.
• Define and implement scalable data architecture using Databricks, cloud platforms, and big data technologies.
• Lead solution design discussions and translate business requirements into robust technical architecture.
• Establish delivery frameworks including Agile execution, release planning, and risk management.
• Provide leadership to globally distributed teams and ensure alignment across engineering, QA, and business teams.
• Drive stakeholder communication with clear reporting on delivery progress, risks, and mitigation plans.
• Promote engineering excellence, code quality, and performance optimization practices across teams.

Data Engineering & Databricks Expertise (Primary Skill)

• Strong hands-on experience with Databricks for building and managing data pipelines and workflows.
• Expertise in Spark (Python/Scala) and distributed data processing frameworks.
• Experience designing and implementing ETL/ELT pipelines on cloud platforms (AWS/Azure).
• Deep understanding of data lakehouse architecture, Delta Lake, and modern data platforms.
• Ability to optimize data processing performance, scalability, and cost-efficiency.
• Experience in data governance, lineage, metadata management, and regulatory compliance requirements.

Architecture & Design Skills

• Proven experience in designing enterprise-scale data architecture and solutions.
• Strong understanding of data modeling (conceptual, logical, physical) and integration design patterns.
• Experience defining architecture standards, best practices, and design governance frameworks.
• Ability to evaluate and recommend modern data tools and frameworks based on business needs.
• Experience integrating on-premise systems with cloud-based data platforms.

Core / Must Have Skills

• 15+ years of experience in technology delivery with strong leadership exposure.
• Extensive experience in Data Engineering and Big Data ecosystems (Hadoop, Spark, Databricks).
• Strong programming expertise in Python/Scala and SQL.
• Experience with cloud platforms such as AWS or Azure.
• Strong understanding of CI/CD, DevOps, and data pipeline automation.
• Excellent communication, stakeholder management, and leadership skills.

Good to Have Skills

• Experience with Kafka or other streaming technologies.
• Knowledge of containerization tools such as Docker and Kubernetes.
• Exposure to financial services or regulatory reporting environments.
• Experience working in global, distributed teams across time zones.

Education & Experience

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• Proven experience leading large-scale transformation or data modernization programs.

About State Street

• Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability.
• We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.
• We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential.
• As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most.
• Join us in shaping the future.
• As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.