Senior Databricks Engineer, Apache Spark and AWS - Vice President

Citibank (Switzerland) AG; Pune District; On-site; Confidential; Full time

Job summary

Citibank (Switzerland) AG in Pune, India seeks an experienced Senior Data Engineer to modernize Spark pipelines on Databricks, migrate from Hadoop, and optimize data processing on AWS. You will design scalable data models, contribute to reusable components, and collaborate with global teams to ensure production-ready solutions.

The role emphasizes translation of architecture into detailed designs, performance tuning, and adherence to engineering standards in a fast-paced environment.

Qualifications

• 10+ years in data engineering or distributed systems.
• Strong expertise in Apache Spark (JavaSpark / PySpark), Databricks on AWS, and Delta Lake.
• Experience on SQL and AWS services.
• Modernization & optimization experience.
• Ability to translate architecture into implementable designs.
• Understanding of data modeling and pipeline orchestration patterns.

Responsibilities

• Platform Engineering & Modernization: refactor Spark pipelines to Databricks native architectures.
• Eliminate legacy Hadoop dependencies; adopt cloud-native AWS patterns.
• Design & solution engineering: translate high-level architecture into detailed designs.
• Performance optimization: improve Spark job performance and simplify pipelines.
• Engineering standards: follow Databricks/Spark standards; write clean, modular code.
• Collaboration: work with senior architects, platform teams, DevOps; provide guidance.
• Testing & QA: develop unit/integration tests; support deployments and validation.

Skills

Spark (Java/PySpark)

Databricks on AWS

Delta Lake

SQL

Spark performance tuning

Data modeling

Architecture design

Education

Bachelor’s degree

Tools

Databricks

Delta Lake

AWS

Job description

Responsibilities:

• 1. Platform Engineering & Modernization Refactor and modernize existing Spark pipelines to Databricks native architectures
• Eliminate legacy Hadoop dependencies and adopt cloud native AWS patterns
• Enhance and extend existing processing logic using optimized Spark (JavaSpark / PySpark) on Databricks
• 2. Databricks Native Development Build and optimize solutions using Databricks features, including Delta Lake, Databricks Workflows for orchestration and Auto scaling and job clusters
• 3. Design & Solution Engineering Contribute to low and mid level architecture and design Translate high level architecture into detailed technical designs Define data models, pipeline patterns, and reusable components Ensure solutions are scalable, maintainable, and production ready
• 4. Performance Optimization & Simplification Analyze, improve Spark job performance and simplify complex or over engineered pipelines into standardized, efficient patterns
• 5. Engineering Standards & Best Practices Follow and contribute to Databricks and Spark engineering standards Write clean, modular, and testable code Contribute to shared frameworks, reusable libraries, and quality standards
• 6. Collaboration & Stakeholder Engagement Work closely with senior architects, platform teams, and DevOps engineers Provide technical inputs, troubleshooting support, and implementation guidance Participate in design discussions and technical decision making
• 7. Testing & Quality Assurance Develop unit, integration, and data validation tests Support production releases and post deployment validation

Qualifications:

• Core Technical Skills 10+ years in data engineering or distributed systems
• Strong expertise in Apache Spark (JavaSpark / PySpark), Databricks on AWS, and Delta Lake
• Experience on SQL
• Experience with AWS services and large-scale distributed data processing
• Modernization & Optimization Experience
• Experience modernizing or refactoring legacy data platforms into cloud-based architectures
• Strong background in Spark performance tuning and large-scale batch optimization
• Design Capability
• Ability to translate architecture into implementable designs
• Understanding of data modeling and pipeline orchestration patterns
• Behavioral Competencies
• Strong problem-solving mindset for complex distributed systems Comfortable working in time-bound, high-impact environments Proactive, accountable, and collaborative Clear communication skills across global teams

Education:

• Bachelor’s degree/University degree or equivalent experience

Job Family Group: Technology

Job Family: Applications Development

Time Type: Full time

Most Relevant Skills Please see the requirements listed above.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi. View Citi’s EEO Policy Statement and the Know Your Rights poster.