Lead Data Engineer (Quantexa)

D L RESOURCES PTE LTD; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

D L RESOURCES PTE LTD in Singapore seeks a Lead Data Engineer to design, build, and optimize our data infrastructure to support advanced analytics for the bank. You will lead a team of data engineers and collaborate with analysts, DevOps, and infrastructure teams to deliver scalable data solutions using Spark, Elasticsearch, and OpenShift.

The role requires a Quantexa certification, 10+ years in data engineering, and strong skills in Spark, Scala, Python, and Kubernetes.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related field.
• At least 10 years of experience as a Data Engineer in large-scale environments with Hadoop, Spark, and data processing.
• Must be Quantexa certified data engineer / data architect and proficient with the tool.
• Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Ranger, etc)
• Experience with OpenShift Container Platform (OCP) and Kubernetes.
• Proficiency in Spark, Python, Scala, or Java.
• Knowledge of DevOps, CI/CD pipelines, and automation tools (Docker, Jenkins, Ansible, Bitbucket).
• Experience with Grafana, Prometheus, Splunk is a plus.
• Excellent collaboration and communication skills; able to lead and mentor.

Responsibilities

• Design, develop, and implement Spark Scala applications and data processing pipelines for large data volumes.
• Integrate Elasticsearch with Spark for efficient indexing and retrieval.
• Tune Spark jobs for performance and scale; ensure data processing efficiency.
• Collaborate with engineers, data scientists, and stakeholders to translate requirements into solutions.
• Design and deploy data engineering solutions on OpenShift Container Platform using containers.
• Optimize workflows for containerized deployment and resource utilization.
• Work with DevOps to streamline deployments, CI/CD, and platform stability.
• Monitor data pipelines, troubleshoot, and implement enhancements.
• Implement monitoring and logging for data infra health and performance.
• Document processes and configurations for knowledge sharing.
• Stay updated on emerging data engineering and DevOps practices.
• Provide technical leadership, mentorship, and continuous improvement of analytics capabilities.

Skills

Quantexa certification

Education

Bachelor's degree in Computer Science or related field

Tools

Hadoop

Spark

OpenShift (OCP)

Kubernetes

Docker

Jenkins

Ansible

Bitbucket

Elasticsearch

Grafana

Prometheus

Splunk

Job description

Key Skills:

Quantexa certification mandatory

Job Objectives

We seek individuals with highly developed conceptual, strategic, and analytical skills, capable of striking a balance between visionary thinking and practical solutions. The ability to comprehend, inspire, and mobilize others is crucial. A business-oriented mindset coupled with effective storytelling will drive your success. We are looking for self-starters ready to take on responsibilities with enthusiasm.

Key Responsibilities

As a Lead Data Engineer, you will play a leading role in designing, building, and optimizing our data infrastructure, ensuring that it supports the advanced analytics need of the bank. You will oversee a team of data engineers, working closely with data analysts, DevOps team, infrastructure engineers, and other stakeholders to deliver high-quality data solution. You will be working with Quantexa platform.

Your main responsibilities will include:

• Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data.
• Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data.
• Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch.
• Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions.
• Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques.
• Optimize data engineering workflows for containerized deployment and efficient resource utilization.
• Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability.
• Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements.
• Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure.
• Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference.
• Stay updated with emerging technologies, industry trends, and best practices in data engineering and DevOps.
• Provide technical leadership, mentorship, and guidance to junior team members to foster a culture of continuous learning and innovation to the continuous improvement of the analytics capabilities within the bank.

Key Requirements

• Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field.
• At least 10 years of experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments.
• Must be Quantexa certified data engineer / data architect and proficient with the tool.
• Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Ranger, etc)
• Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes.
• Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java.
• Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
• Experience with Grafana, Prometheus, Splunk will be an added benefit
• Strong problem-solving and troubleshooting skills with a proactive approach to resolving technical challenges.
• Excellent collaboration and communication skills to work effectively with cross-functional teams.
• Ability to manage multiple priorities, meet deadlines, and deliver high-quality results in a fast-paced environment.
• Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services is a plus.