Python and Apache Spark Big Data Engineer - Assistant Vice President

Mployee.me; Pune District; On-site; INR 2,600,000 - 4,200,000; Full time

Job summary

Citi in Pune, India is seeking an experienced Big Data Engineer to design, build and optimize data processing pipelines across Hadoop/Spark ecosystems. You will collaborate with global teams to deliver scalable, reliable analytics solutions.

The role requires 8–12 years of hands-on experience, strong Python/Scala, SQL, and SDLC discipline, with a focus on performance tuning, low-latency components and data quality across large data volumes.

Qualifications

• 8–12 years of experience in big data and analytics.
• Proficient in Python and/or Spark.
• Experience with HDFS, MapReduce, YARN, Spark, Hive.
• Familiar with Avro, Parquet, CSV, JSON.
• Strong SDLC, unit testing, and architecture fundamentals.
• Unix/Linux and shell scripting experience.
• SQL knowledge and DB principles.
• Agile/Scrum experience.
• GIT/JIRA SCM/tools.
• Kafka real-time processing.

Responsibilities

• Translate application storyboards and use cases into functional applications.
• Design, build, and maintain efficient, reusable, and reliable Python/Scala code.
• Ensure the best possible performance, quality, and responsiveness of the applications.
• Identify bottlenecks and bugs, and devise solutions to these problems.
• Develop high performance & low latency components to run Spark clusters.
• Interpreting functional requirements into design approaches those can be served through Big Data platform.
• Collaborate and partner with Global Teams based across different locations.
• Be able to propose best practices and standards.
• Perform the testing of software prototypes and transfer to the operational team.
• Processing of data using Hive, Impala & HBASE.
• Performing analysis of large data sets and derive insights.

Skills

Python

Spark

Big Data

SQL

UNIX/Linux

Agile/Scrum

GIT

JIRA

Kafka

Hive

Data formats

Tools

HDFS

MapReduce

YARN

Hive

Impala

HBase

Maven

Jenkins

TeamCity

Job description

Experience Range - 8 to 12 years

Strong experience with one or more programming languages (Python/Spark).

• Demonstrated analytical and problem-solving skills. Application performance tuning, troubleshooting experience and the implementation of these skills in Big Data domain.
• Comfortable working with large data volumes and be able to demonstrate a firm understanding of logical data structures and analysis techniques.
• Experience in Big data technologies like HDFS, Map Reduce, YARN, Apache Spark, Hive.
• Familiarity with data formats like Avro, Parquet, CSV, JSON.
• Strong systems analysis, design and architecture fundamentals, Unit Testing and other SDLC activities.
• Understanding fundamental design principles behind a scalable application.
• Experience in working with UNIX/LINUX and shell scripting.
• Good knowledge of database principles, practices with SQL.
• Application performance tuning, troubleshooting experience and the implementation of these skills in Big Data domain.
• Agile/Scrum methodology experience is required.
• Experience in SCMs like GIT and tools like JIRA.
• Experience in No SQL databases.
• Familiarity with build tools such as Maven and continuous integration like Jenkins/Team City.
• Familiarity with cloud and container technologies.
• Experience real time data processing - Kafka.

Responsibilities

• Translate application storyboards and use cases into functional applications.
• Design, build, and maintain efficient, reusable, and reliable Python/Scala code.
• Ensure the best possible performance, quality, and responsiveness of the applications.
• Identify bottlenecks and bugs, and devise solutions to these problems.
• Develop high performance & low latency components to run Spark clusters.
• Interpreting functional requirements into design approaches those can be served through Big Data platform.
• Collaborate and partner with Global Teams based across different locations.
• Be able to propose best practices and standards.
• Perform the testing of software prototypes and transfer to the operational team.
• Processing of data using Hive, Impala & HBASE.
• Performing analysis of large data sets and derive insights.

Job Family Group:

Technology

Job Family:

Applications Development

Time Type:

Full time

Most Relevant Skills

Please see the requirements listed above.

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.