Data Engineer - (PySpark+Hadoop)

Krisvconsulting Services Pte Ltd; Singapore; On-site; SGD 70,000 - 100,000; Full time

Job summary

A consulting services company in Singapore is looking for a skilled Data Engineer with 4-10 years of experience in Hadoop and Spark technologies. The ideal candidate will create data transformation jobs, work with Spark SQL, and produce unit tests for Spark transformations. Knowledge in core banking and finance, along with familiarity in cloud platforms like AWS, is preferred. Join us to ensure effective and timely data solutions in a collaborative environment.

Qualifications

• 4-10 years of experience as a Hadoop Data Engineer.
• Strong expertise in Hadoop, Spark, Scala, PySpark, Python, Hive, and Impala.
• Experience in core banking and finance domains.
• Good knowledge of data warehousing methodologies.

Responsibilities

• Create Spark Scala/PySpark jobs for data transformation.
• Produce unit tests for Spark transformations.
• Use Spark and Spark-SQL for data reading and table creation.
• Prepare design and operations documentation.
• Perform peer code quality reviews.
• Engage in hands-on coding in a pair programming environment.

Skills

Hadoop

Spark

PySpark

Scala

Python

Hive

CI/CD

Git

Jenkins

Agile Methodologies

DevOps

Oracle

Kafka

Machine Learning

AWS (Cloud)

Job description

About the job Data Engineer - (PySpark+Hadoop)

Key Responsibilities:

• Create Spark Scala/PySpark jobs for data transformation and aggregation.
• Produce unit tests for Spark transformations and helper methods.
• Use Spark and Spark-SQL to read parquet data and create tables in Hive using the Scala API.
• Work closely with Business Analysts to review test results and obtain sign-off.
• Prepare necessary design and operations documentation for future use.
• Perform peer code quality reviews and ensure compliance with quality standards.
• Engage in hands-on coding, often in a pair programming environment.
• Collaborate with teams to build quality code and ensure smooth production deployments.

Requirements:

• 4-10 years of experience as a Hadoop Data Engineer, with strong expertise in Hadoop, Spark, Scala, PySpark, Python, Hive, Impala, CI/CD, Git, Jenkins, Agile Methodologies, DevOps, and Cloudera Distribution.
• Strong knowledge of data warehousing methodologies.
• Minimum of 4 years of relevant experience in Hadoop and Spark/PySpark.
• Strong understanding of enterprise data architectures and data models.
• Experience in the core banking and finance domains.
• Familiarity with Oracle, Spark streaming, Kafka, and machine learning.
• Good to have cloud experience, particularly with AWS.
• Ability to develop applications using the Hadoop tech stack efficiently and effectively, ensuring on-time, in-specification, and cost-effective delivery.