Data Engineering Developer - Assistant Manager

State Street Corporation; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

State Street Corporation in Bengaluru is seeking a Data Engineer to design, build, and optimize large-scale data pipelines using Databricks, Apache Spark, and Scala.

The role emphasizes cloud-native solutions on Microsoft Azure, data modeling, and robust ETL/ELT frameworks. You will work in an Agile, collaborative environment delivering high-quality data products.

Qualifications

• 10+ years of experience in Data Engineering.
• Hands-on Databricks, Spark, Scala development.
• Strong cloud experience with Microsoft Azure.
• Proven ability to design scalable data architectures.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Develop and optimize Spark jobs for performance.
• Implement Databricks workflows and orchestration.
• Troubleshoot production issues and bottlenecks.
• Collaborate with cross-functional teams on data architectures.
• Follow CI/CD and best practices in development.

Skills

Databricks

Apache Spark

Scala

Spark SQL

PySpark

ETL/ELT pipelines

Data modeling & warehousing

Azure cloud

SQL development

Git/GitHub

Education

Bachelor's or Master's in CS/Engineering

Job description

Role Summary We are seeking a highly skilled Data Engineer with strong expertise in Databricks, Apache Spark, and Scala to design, develop, and optimize large-scale data processing solutions. The ideal candidate will have hands-on experience building distributed data pipelines, implementing scalable ETL frameworks, and delivering cloud-native data solutions on Azure.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Scala.
• Build ingestion, transformation, and validation frameworks for high-volume data processing.
• Develop and optimize Spark jobs for performance, scalability, and reliability.
• Implement and manage Databricks workflows, job orchestration, and scheduling.
• Troubleshoot production issues, performance bottlenecks, and pipeline failures.
• Optimize cluster utilization, Spark execution plans, and data processing efficiency.
• Work with cross-functional teams to design data architectures and integration solutions.
• Implement best practices for coding, testing, CI/CD, and deployment.
• Support production environments and participate in incident resolution and root cause analysis.
• Collaborate in Agile teams to deliver high-quality data products.

Required Skills

• Strong hands-on experience in: Databricks Apache Spark Scala Spark SQL PySpark
• Experience in: Distributed data processing frameworks ETL/ELT pipeline development Data modeling and data warehousing concepts Performance tuning and optimization of Spark applications
• Cloud experience: Microsoft Azure Azure Data Lake Storage (ADLS) Azure Data Factory (ADF)
• Strong SQL development and query optimization skills.
• Experience with source control systems such as Git/GitHub.
• Strong analytical and debugging skills.

Preferred Skills

• Delta Lake Unity Catalog CI/CD pipelines (Azure DevOps, Harness)
• Snowflake Data Quality and Validation Frameworks Financial Services / Reference Data domain knowledge
• Experience with AI-assisted development tools such as GitHub Copilot

Education & Experience

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 10+ years of experience in Data Engineering.
• 6+ years of hands-on Databricks and Spark development experience.
• Strong experience developing production-grade Scala applications.

Nice-to-Have

• Databricks Certification
• Azure Certification
• Experience with Real-Time Streaming (Kafka, Spark Structured Streaming)
• Exposure to Lakehouse architecture patterns

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success. We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at StateStreet.com/careers Read our CEO Statement

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success. We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.