Senior Data Engineer - Apache Spark and SQL - Vice President

Citi; Pune District; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

Citi in Pune seeks a Senior Databricks Engineer to drive modernization of data platforms on Databricks on AWS, migrating from Cloudera Hadoop and shaping scalable data pipelines for production workloads. You will refactor Spark jobs, apply Delta Lake and Databricks Workflows, collaborate with architects and DevOps, and ensure robust, maintainable solutions across global teams.

You will be immersed in performance tuning, testing, and production readiness while advancing the data platform’s

Qualifications

• 11+ years in data engineering or distributed systems.
• Strong expertise in Apache Spark (JavaSpark / PySpark), Databricks on AWS, and Delta Lake.
• Experience with SQL and large-scale distributed data processing.

Responsibilities

• Refactor Spark pipelines to Databricks native architectures.
• Eliminate legacy Hadoop dependencies and adopt cloud-native patterns.
• Build and optimize solutions using Delta Lake and Databricks Workloads for orchestration.
• Ensure solutions are scalable, maintainable, and production ready.
• Collaborate with senior architects and DevOps teams.

Skills

Databricks on AWS

Apache Spark

Delta Lake

SQL

AWS services

Distributed processing

Education

Bachelor’s degree/University degree or equivalent experience

Job description

We are looking for a highly skilled Senior Databricks Engineer to contribute to the engineering, modernization, and continuous evolution of data processing platform on Databricks on AWS. While supporting the transition from the legacy Cloudera Hadoop platform to Databricks on AWS, this role will continue to play a key part in enhancing performance, simplifying pipelines, and delivering new capabilities on the Databricks platform over the long term.

Responsibilities

Platform Engineering & Modernization

• Refactor and modernize existing Spark pipelines to Databricks native architectures
• Eliminate legacy Hadoop dependencies and adopt cloud native AWS patterns
• Enhance and extend existing processing logic using optimized Spark (JavaSpark / PySpark) on Databricks

Databricks Native Development

• Build and optimize solutions using Databricks features, including Delta Lake, Databricks Workflows for orchestration and Auto scaling and job clusters

Design & Solution Engineering

• Contribute to low and mid level architecture and design
• Translate high level architecture into detailed technical designs
• Define data models, pipeline patterns, and reusable components
• Ensure solutions are scalable, maintainable, and production ready

Performance Optimization & Simplification

• Analyze, improve Spark job performance and simplify complex or over engineered pipelines into standardized, efficient patterns

Engineering Standards & Best Practices

• Follow and contribute to Databricks and Spark engineering standards
• Write clean, modular, and testable code
• Contribute to shared frameworks, reusable libraries, and quality standards

Collaboration & Stakeholder Engagement

• Work closely with senior architects, platform teams, and DevOps engineers
• Provide technical inputs, troubleshooting support, and implementation guidance
• Participate in design discussions and technical decision making

Testing & Quality Assurance

• Develop unit, integration, and data validation tests
• Support production releases and post deployment validation

Qualifications

Core Technical Skills

• 11+ years in data engineering or distributed systems
• Strong expertise in Apache Spark (JavaSpark / PySpark), Databricks on AWS, and Delta Lake
• Experience on SQL
• Experience with AWS services and large‑scale distributed data processing

Modernization & Optimization Experience

• Experience modernizing or refactoring legacy data platforms into cloud‑based architectures
• Strong background in Spark performance tuning and large‑scale batch optimization

Design Capability

• Ability to translate architecture into implementable designs
• Understanding of data modeling and pipeline orchestration patterns

Behavioral Competencies

• Strong problem‑solving mindset for complex distributed systems
• Comfortable working in time‑bound, high‑impact environments
• Proactive, accountable, and collaborative
• Clear communication skills across global teams

Education

• Bachelor’s degree/University degree or equivalent experience

Job Family Group

Technology

Job Family

Applications Development

Time Type

Full time

Most Relevant Skills

Please see the requirements listed above.

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.