Cloud Data Engineer: Real-Time Pipelines & Analytics

NCS Group; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

NCS Group in Singapore is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms. You will integrate diverse data sources, support analytics, reporting, and machine learning workloads, and collaborate with analytics, product, and infrastructure teams.

You will implement governance, monitoring, and reliability best practices, optimize ETL performance, and build dashboards and data visualizations while advancing our cloud-based data

Qualifications

• 3+ years of experience in data engineering with scalable pipelines.
• Experience designing data solutions including data modelling and distributed computing architectures.
• Hands-on experience with PySpark, Spark SQL, and Databricks notebooks/jobs.
• Experience orchestrating data pipelines with ADF, Airflow, or similar tools.
• Experience with real-time and batch data processing.
• Experience building pipelines on Azure, with AWS experience beneficial.
• Proficiency in SQL including window functions and performance optimization.
• Understanding of DevOps tools, Git workflows, and CI/CD pipelines.
• Familiarity with Scrum methodology and experience working in Scrum teams.
• Ability to apply Scrum practices in a practical project context.
• Strong problem-solving and collaborative mindset.
• Experience with streaming technologies such as Apache Kafka, Apache Flink, or AWS Kinesis.
• Ability to design and implement real-time data processing pipelines.

Responsibilities

• Develop and maintain ETL pipelines for centralized data storage systems (e.g. Delta Lake).
• Integrate data from databases, APIs, log files, streaming platforms, and external providers
• Develop data transformation routines to clean, normalize, and aggregate data
• Apply data processing techniques to handle complex or inconsistent datasets
• Contribute to frameworks and best practices for code development and deployment
• Implement data governance in alignment with company standards
• Partner with analytics and product leaders to design and operationalize pipelines
• Collaborate with infrastructure leaders to advance cloud-based data platforms
• Explore new tools and techniques leveraging Azure, Databricks, or related platforms
• Monitor data pipelines to detect and resolve issues promptly
• Develop monitoring tools, alerts, and automated error-handling mechanisms
• Analyse business requirements and identify data extraction requirements
• Attend requirement grooming and refinement sessions with users
• Develop and maintain ETL pipelines for ingestion, transformation, validation, and loading
• Optimise performance and batch scheduling
• Develop dashboards, reports, scorecards, and data visualizations
• Perform SIT, data validation, data profiling and confirm data accuracy
• Validate completeness and consistency of ETL Loads
• Support UAT and production implementation

Skills

Problem-solving

Collaborative mindset

Scrum practices

Data engineering mindset

Education

Databricks Certified Data Engineer Associate

Databricks Certified Data Engineer Professional

Tools

Databricks

PySpark

Spark SQL

Databricks notebooks

Azure

AWS

ADF

Airflow

SQL

Git workflows

CI/CD pipelines

Apache Kafka

Apache Flink

AWS Kinesis

Job description

NCS Group in Singapore is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms. You will integrate diverse data sources, support analytics, reporting, and machine learning workloads, and collaborate with analytics, product, and infrastructure teams.

You will implement governance, monitoring, and reliability best practices, optimize ETL performance, and build dashboards and data visualizations while advancing our cloud-based data