Big Data Engineer

GECO Asia Pte Ltd; Petaling Jaya; On-site; MYR 120,000 - 160,000; Full time

Job summary

An innovative data solutions company is seeking a highly skilled Big Data Engineer with over 8 years of experience in data migration, data setup, and data systems development. The ideal candidate will have deep expertise in Apache Spark, SQL, and Java (with Scala) for large-scale data processing. This role will involve designing and optimizing ETL pipelines, writing complex SQL queries, and developing backend applications. Strong knowledge of data architecture and experience with regression testing are essential. Competitive salary and benefits offered.

Qualifications

• 8+ years of experience in data engineering, with a focus on big data technologies.
• Strong proficiency in Apache Spark, SQL, and Java/Scala.
• Hands-on experience with regression testing and cutover planning in large-scale data migrations.

Responsibilities

• Design, develop, and optimize Spark-based ETL pipelines for large-scale data processing.
• Write and optimize complex SQL queries for data extraction and transformation.
• Develop backend services and data processing applications using Java and Scala.

Skills

Apache Spark

SQL

Java

Scala

Data migration

Data architecture

Regression testing

Cloud platforms

Problem-solving

Communication

Tools

Hadoop

Docker

Kubernetes

Job description

Job Summary:

We are seeking a highly skilled Big Data Engineer with 8 years of experience in data migration, data setup, and data systems development. The ideal candidate will have deep expertise in Apache Spark, SQL, and Java (with Scala) for large-scale data processing, reporting, and system development. Strong knowledge of data architecture, semantic layer development, and experience in regression testing and cutover activities for enterprise-level migrations is essential.

Key Responsibilities:

Spark:

• Design, develop, and optimize Spark-based ETL pipelines for large-scale data processing and analytics.
• Utilize Spark SQL, DataFrames, RDDs, and Streaming for efficient data transformations.
• Tune Spark jobs for performance, including memory management, partitioning, and execution plans.
• Implement real-time and batch data processing using Spark Streaming or Structured Streaming.

SQL:

• Write and optimize complex SQL queries for data extraction, transformation, and aggregation.
• Perform query performance tuning, indexing, and partitioning for efficient execution.
• Develop stored procedures, functions, and views to support data operations.
• Ensure data consistency, integrity, and security across relational databases.

Java (Preferred with Scala Knowledge):

• Develop backend services and data processing applications using Java and Scala.
• Optimize JVM performance, including memory management and garbage collection, for Spark workloads.
• Leverage Scala’s functional programming capabilities for efficient data transformations.
• Implement multithreading, concurrency, and parallel processing in Java for high-performance systems.

Required Skills & Qualifications:

• 8+ years of experience in data engineering, with a focus on big data technologies.
• Strong proficiency in Apache Spark, SQL, and Java/Scala.
• Experience in data migration, data setup, and semantic layer development.
• Solid understanding of data architecture, ETL frameworks, and data governance.
• Hands-on experience with regression testing and cutover planning in large-scale data migrations.
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) is a plus.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Preferred Qualifications:

• Experience with Hadoop ecosystem tools (Hive, HDFS, Oozie, etc.).
• Knowledge of containerization and orchestration (Docker, Kubernetes).
• Exposure to CI/CD pipelines and DevOps practices.
• Relevant certifications in Big Data or Cloud technologies.