Senior Data Engineer — Lead Reliable Data Pipelines

THOUGHTWORKS PTE. LTD.; Singapore; On-site; SGD 180,000 - 280,000; Full time

Job summary

Thoughtworks PTE. LTD. is seeking a Lead Data Engineer to provide technical leadership, incident management expertise, and strategic oversight for reliable, secure, and scalable data platforms.

You will define best practices, influence stakeholders, and mentor the team through complex technical challenges. You will diagnose incidents, monitor data pipelines, enforce data quality, and drive continuous delivery for high‑quality data solutions while collaborating with cross‑functional teams and

Qualifications

• Experience with data projects, including data frameworks, ETL processes, and data visualization.
• Skilled in working with data models and pipelines to support smooth and efficient operations.
• Proficient in MSSQL databases and AWS cloud-native services.
• Strong hands-on experience with Python and Shell scripting.
• Possess strong debugging and triaging skills to troubleshoot data issues effectively.
• Experience building and supporting APIs within microservice architectures.
• Familiar with data processing frameworks such as Apache Spark and Hadoop.
• Practical experience with data science tools, including AWS SageMaker.
• Comfortable working in Agile environments, including Scrum and/or Kanban.
• (Desirable) Exposure to Java (Spring Boot), frontend frameworks like Angular/React, and knowledge of architectural styles such as monolithic, layered, microservices, and serverless.

Responsibilities

• Work closely with cross‑functional teams, including client systems and data engineers, to diagnose and resolve incidents impacting data operations, ensuring minimal business disruption.
• Monitor, troubleshoot, and optimize data pipelines to ensure reliable and timely data delivery, addressing any issues encountered in production environments.
• Use incident management tools, data monitoring metrics, and reports to detect issues and take corrective actions.
• Enforce and monitor data quality metrics to maintain consistency, accuracy, and completeness across systems.
• Leverage logging techniques for alerting, monitoring, and identifying root causes in data platforms and pipelines.
• Optimize and modify data pipelines for scalability and performance improvements.
• Support deployment of new features, bug fixes, and enhancements, ensuring smooth production integration and minimal risk.
• Maintain documentation (troubleshooting guides, workflows, best practices) and mentor junior peers through technical and leadership guidance.
• Follow standards and best practices to enhance operational efficiency, system stability, and availability.
• Apply continuous delivery practices to support and deliver high‑quality data solutions and customer value early and often.

Skills

Data projects

Data models

Data pipelines

MSSQL

AWS

Python

Shell scripting

Debugging

APIs

Spark

Hadoop

SageMaker

Agile

Java

Spring Boot

Angular/React

Tools

MSSQL

AWS

Spark

Hadoop

SageMaker

Spring Boot

Angular

React

Job description

Thoughtworks PTE. LTD. is seeking a Lead Data Engineer to provide technical leadership, incident management expertise, and strategic oversight for reliable, secure, and scalable data platforms.

You will define best practices, influence stakeholders, and mentor the team through complex technical challenges. You will diagnose incidents, monitor data pipelines, enforce data quality, and drive continuous delivery for high‑quality data solutions while collaborating with cross‑functional teams and