Python and Apache Spark Big Data Engineer - Assistant Vice President

Citi; Pune District; Sur place; INR 2 800 000 - 4 200 000; Plein temps

Résumé du poste

Citi in Pune, India seeks a senior Big Data Engineer to transform requirements into scalable data applications. You will develop and optimize Python/Scala code for Spark clusters, ensure performance, and troubleshoot bottlenecks within large data pipelines.

The role requires 8–12 years in the Big Data domain, strong Kafka experience, and hands-on with HDFS, Hive, and related technologies. Collaboration with global teams and adherence to Agile practices are essential.

Qualifications

• 8–12 years of experience required in Big Data
• Strong experience with Python and Spark
• Proficient in real-time data processing with Kafka
• Experience with HDFS, MapReduce, YARN, Hive and Spark ecosystems
• Familiarity with data formats Avro/Parquet/CSV/JSON
• Solid SDLC and Agile/Scrum background
• Unix/Linux scripting and SQL/noSQL databases
• Version control with GIT and issue tracking with JIRA
• Build and CI/CD with Maven and Jenkins/TeamCity
• Cloud and container technologies awareness

Responsabilités

• Translate requirements into scalable data applications
• Develop and maintain high-performance Python/Scala code for Spark clusters
• Ensure optimal performance, quality, and responsiveness
• Identify bottlenecks and implement fixes for big data pipelines
• Design high-throughput, low-latency data processing components
• Collaborate with global teams and establish best practices
• Perform testing and handover to operations
• Process data using Hive/Impala/HBase and analyze large datasets

Connaissances

Python/Scala

Spark

Data analysis

Problem solving

SDLC

Agile/Scrum

Unix/Linux

SQL

NoSQL

GIT/JIRA

Cloud/Containers

Kafka

Hadoop

Outils

HDFS

MapReduce

YARN

Hive

Parquet/Avro/CSV/JSON

Maven

Jenkins/TeamCity

Description du poste

• Experience real time data processing - Kafka.

Experience Range - 8 to 12 years

Strong experience with one or more programming languages (Python/Spark).

• Demonstrated analytical and problem-solving skills. Application performance tuning, troubleshooting experience and the implementation of these skills in Big Data domain.
• Comfortable working with large data volumes and be able to demonstrate a firm understanding of logical data structures and analysis techniques.
• Experience in Big data technologies like HDFS, Map Reduce, YARN, Apache Spark, Hive.
• Familiarity with data formats like Avro, Parquet, CSV, JSON.
• Strong systems analysis, design and architecture fundamentals, Unit Testing and other SDLC activities.
• Understanding fundamental design principles behind a scalable application.
• Experience in working with UNIX/LINUX and shell scripting.
• Good knowledge of database principles, practices with SQL.
• Application performance tuning, troubleshooting experience and the implementation of these skills in Big Data domain.
• Agile/Scrum methodology experience is required.
• Experience in SCMs like GIT and tools like JIRA.
• Experience in No SQL databases.
• Familiarity with build tools such as Maven and continuous integration like Jenkins/Team City.
• Familiarity with cloud and container technologies.
• Experience real time data processing - Kafka.

Responsibilities:

• Translate application storyboards and use cases into functional applications.
• Design, build, and maintain efficient, reusable, and reliable Python/Scala code.
• Ensure the best possible performance, quality, and responsiveness of the applications.
• Identify bottlenecks and bugs, and devise solutions to these problems.
• Develop high performance & low latency components to run Spark clusters.
• Interpreting functional requirements into design approaches those can be served through Big Data platform.
• Collaborate and partner with Global Teams based across different locations.
• Be able to propose best practices and standards.
• Perform the testing of software prototypes and transfer to the operational team.
• Processing of data using Hive, Impala & HBASE.
• Performing analysis of large data sets and derive insights.

Job Family Group:

Technology

Job Family:

Applications Development

Time Type:

Full time

Most Relevant Skills

Please see the requirements listed above.

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.