Senior Data Engineer (Hive SQL, Hadoop, Cloudera)

NEPTUNEZ SINGAPORE PTE. LTD.; Singapore; On-site; SGD 70,000 - 90,000; Full time

Job summary

A leading data solutions company in Singapore is looking for a skilled Data Engineer to manage and migrate data from MariaDB to Cloudera. The ideal candidate will have over 5 years of experience in data engineering, proficient in SQL, Hive SQL, and Spark. Responsibilities include data modeling, optimizing data processing, and collaborating with cross-functional teams to ensure data integrity. Join us to help shape our big data landscape in a vibrant technological environment.

Qualifications

• 5+ years of experience in data engineering and migration projects for big data.
• Strong hands-on experience on Cloudera and its components.
• Highly proficient in SQL, Hive SQL, Spark, and data modeling.

Responsibilities

• Migrate existing data warehouse from MariaDB to Hadoop Big Data Cloudera.
• Develop and optimize SQL, Hive SQL, and Spark scripts.
• Collaborate with teams to understand data requirements.
• Document processes, procedures, and best practices.

Skills

Data engineering

SQL

Hive SQL

Spark

ETL processes

Data modeling

Version control (Bitbucket, GIT)

Scheduling jobs with Autosys

Communication

Tools

Cloudera

Dataiku

Job description

Responsibilities

• Migrate existing data warehouse and data model from MariaDB to Hadoop Big Data Cloudera on-premises platform.
• Develop and optimize SQL, Hive SQL, and Spark scripts to ensure efficient data processing.
• Design and implement data models to support business requirements and optimize performance.
• Collaborate with cross-functional teams to understand data requirements and ensure data integrity throughout the migration process.
• Develop and execute test plans to validate data accuracy and system performance.
• Coordinate with stakeholders (internal) to plan and execute production deployments.
• Schedule and monitor jobs using Autosys to ensure timely execution and minimize downtime.
• Provide technical expertise and guidance to team members throughout the migration project.
• Document processes, procedures, technical specifications, and best practices to facilitate knowledge sharing and ensure scalability.
• Create and maintain unit test case documents to ensure code quality and reliability.

Requirements

• Proven experience of 5+ years in data engineering and migration projects for big data (Hortonworks / Cloudera)
• Strong hands‑on experience on Cloudera and related ecosystem components.
• Strong experience in implementing ETL (Extract, Transform, Load) processes.
• Highly proficient in SQL, Hive SQL, Spark, and data modeling.
• Strong understanding of the production deployment process.
• Experience with scheduling jobs using Autosys or similar.
• Experience with version control systems such as Bitbucket, GIT etc.
• Ability to troubleshoot and resolve data related issues efficiently.
• Good communication and interpersonal skills
• Experience with Dataiku is a plus, but not mandatory.