Senior Data Engineer

EXL; Gurugram District; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading data solutions provider seeks an Executive - Recruitment Coordinator in Gurugram, India. This full-time position involves designing and building data solutions using Scala and PySpark, optimizing performance on Hadoop, and collaborating with teams. The ideal candidate will have mid-senior level experience and a passion for data-driven scenarios. As part of your role, you will participate in agile processes while creating solutions that offer analytical insights and enhance operational efficiencies.

Qualifications

• Experience in designing and building solutions using Scala and PySpark.
• Understanding of Hadoop clusters and data flow integration.
• Ability to optimize Spark application performance.

Responsibilities

• Design and build applications on the Spark framework.
• Develop and execute data pipeline testing processes.
• Integrate multiple data sources and ensure data quality.
• Participate in the agile development process.

Skills

Scala

PySpark

Hadoop

Spark Streaming

Unix shell scripting

RDBMS

Agile methodologies

Tools

Apache Kafka

Jenkins

Bitbucket

Git

Job description

Executive - Recruitment Coordinator at EXL Company

Job Description: We are looking for a Scala & PySpark solutions developer and data engineer who can design and build solutions for one of our Fortune 500 client programs. The role focuses on building data standardization and curation solutions on a Hadoop cluster, integrating data across internal and external sources, providing analytical insights, and ensuring integration with the customer’s critical systems.

Key Responsibilities

• Design, build, and unit test applications on the Spark framework using Scala and Python.
• Develop Spark‑based applications for batch and streaming requirements, leveraging Hadoop and NoSQL databases.
• Develop and execute data pipeline testing processes and validate business rules and policies.
• Optimize performance of Spark applications in Hadoop using Spark Context, Spark‑SQL, DataFrame, and Pair RDD configurations.
• Choose appropriate native Hadoop file formats (Avro, Parquet, ORC, etc.) and compression codecs for data access.
• Design and build real‑time applications using Apache Kafka and Spark Streaming.
• Integrate Unix shell scripting, RDBMS, Hive, HDFS File System, and HDFS compression codecs into solutions.
• Build data tokenization libraries and integrate with Hive and Spark for column‑level obfuscation.
• Process large amounts of structured and unstructured data, including integration from multiple sources.
• Create and maintain an integration and regression testing framework on Jenkins, integrated with Bitbucket or Git repositories.
• Participate in the agile development process, documenting and communicating issues and bugs related to data standards in scrum meetings.
• Collaborate with onsite and offshore teams.
• Identify and resolve complex data‑driven scenarios, triage defects, and address production issues.
• Continuously learn and adapt to new concepts with an open, analytical mindset.
• Participate in code release and production deployment.
• Challenge and inspire team members to achieve business results in a fast‑paced, quickly changing environment.

Seniority Level

• Mid‑Senior level

Employment Type

• Full‑time

Job Function

• Business Development