Lead DevOps Engineer

Great Eastern; Singapore; On-site; SGD 120,000 - 160,000; Full time

Job summary

Great Eastern in Singapore is seeking a senior data ingestion engineer to lead end-to-end data ingestion from multiple sources into the data lake. You will design, implement, and optimize pipelines and automations to support analytics and AI initiatives.

The role requires deep experience with Spark, Hadoop, SQL, ETL tools like Talend, and job schedulers such as Autosys or Control-M, plus collaboration with data engineering and IT teams to ensure SLA performance.

Qualifications

• Bachelor or Master Degree in Computer Science, Engineering, or similar relevant field.
• 8+ years of data ingestion/engineering with Hadoop tech stack.
• Proficient with Scala and PySpark.
• Hands-on experience on Spark framework and other distributed data processing frameworks like Hadoop Map-Reduce, Hive etc. Proficient in ETL tools like Talend.
• Proficient in RDBMS databases such as Oracle, MySQL, MSSqlServer.
• Strong scripting skills in Linux environment and SQL.
• Expertise in Hadoop ecosystems.
• Hands-on Experience in Sqoop, Hive, Spark, Python, Scala is a must.
• Hands-on Experience in Job orchestration / Job schedulers like Autosys, Control-M
• Good to have working experience with one of the cloud platforms like AWS, Azure or GCP.
• Ability to plan and organize technical work and deliverables.
• Ability to follow guidelines and adhere to the established software development standards and conventions.
• Self-motivated and independent.
• Able to work with minimum supervision and to work well with stakeholders and project staff.
• Ability to prioritize and multi-task across numerous work streams.
• Strong interpersonal skills; ability to work on cross-functional teams.
• Strong verbal and written communication skills.
• Deep knowledge of best practices across data architectures and data warehousing/BI.
• Demonstrated problem-solving skills and ability to learn and meet deadlines.

Responsibilities

• Analyze data requirements across entities; develop, implement and maintain data ingestion pipelines from source to data lake.
• Automate data validation steps and report generation.
• Automate reusable scripts for analytics and reporting needs.
• Ensure end-to-end data extraction and integration with analytics/reporting systems.
• Design job schedules and configuration files for schedulers.
• Provide production L2 batch support after deployment.
• Perform code reviews for team-developed applications.
• Contribute to initiatives bringing data into the data lake and delivering insights.
• Monitor performance and SLA adherence; optimize ingestion processes.
• Collaborate with data engineering and IT teams.
• Troubleshoot MapReduce/Spark jobs and tune performance in production.
• Develop and sustain technical knowledge, certifications, and skills.
• Handle day-to-day tasks with moderate direction and supervision.

Skills

Data ingestion

Scala

PySpark

Spark

ETL

SQL

Linux scripting

Hadoop ecosystem

Job orchestration

Cloud platforms

Education

Bachelor/Master in CS/Engineering

Tools

Talend

Hive

Sqoop

Autosys

Control-M

AWS

Azure

GCP

Hadoop

Oracle

MySQL

MS SQL Server

Spark

Python

Scala

Job description

This role oversees and manages the ingestion framework to ingest data from various data sources for data analytics and AI purposes. Detailed activities include:

• Analyzing the data requirements across various entities, develop, implement and maintain data ingestion pipelines from source to data lake pipelines.
• Automating data validation steps and report generation.
• Automating codes/ scripts that can be repeatedly used across similar analytics and reporting requirements.
• Managing and ensuring accuracy and timeliness and automation solution in end to end data extraction and integration with analytics and management reporting system.
• Designing the jobs schedule in scheduling tools and developing the configuration files for job schedulers.
• Performing production L2 batch support after production deployment.
• Perform code review functions for applications / programs developed by team members.
• To be part of initiatives that brings data into the data lake and delivers insights.
• Monitor and measure performance to assure ongoing data ingestion is meeting the SLA and optimization of the ingestion process to manage the performance and the SLAs.
• Work effectively with other stakeholders such as data engineering team, IT team, etc.
• Troubleshoot MapReduce/Spark Jobs and do performance tuning in production environments.
• Independently develop and sustain technical knowledge, certifications, and skills.
• Effectively handling day-to-day assignments given moderate directions and supervision.

• Bachelor or Master Degree in Computer Science, Engineering, or similar relevant field.
• Working experience in data ingestion or data engineering with Hadoop tech stack for 8+ years.
• Proficient with Scala and PySpark.
• Hands-on experience on Spark framework and other distributed data processing frameworks like Hadoop Map-Reduce, Hive etc. Proficient in ETL tools like Talend.
• Proficient in RDBMS databases such as Oracle, MySQL, MSSqlServer.
• Strong scripting skills in Linux environment and SQL.
• Expertise in Hadoop ecosystems.
• Hands-on Experience in Sqoop, Hive, Spark, Python, Scala is a must.
• Hands-on Experience in Job orchestration / Job schedulers like Autosys, Control-M
• Good to have working experience with one of the cloud platforms like AWS (Amazon Web Services), Microsoft Azure or Google Cloud Platform.
• Ability to plan and organize technical work and deliverables.

• Ability to follow guidelines and adhere to the established software development standards and conventions.

• Self-motivated and independent.

• Able to work with minimum supervision and to work well with stakeholders and project staff.

• Ability to prioritize and multi-task across numerous work streams.

• Strong interpersonal skills; ability to work on cross-functional teams.

• Strong verbal and written communication skills.
• Deep knowledge of best practices through relevant experience across data-related disciplines and technologies particularly for enterprise wide data architectures and data warehousing/BI.
• Demonstrated problem-solving skills. Ability to learn effectively and meet deadline.