Databricks Data Engineer: Scalable Pipelines & Analytics

NCS Group; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

NCS Group is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms in Singapore. You will integrate diverse data sources, ensure data quality, and support analytics, reporting, and ML workloads while aligning with governance and reliability standards.

The role collaborates with analytics, product, and infrastructure teams to enhance the data platform, implement real-time and batch processing, and advance cloud-based capabilities using

Qualifications

• 3+ years of experience in data engineering with scalable pipelines.
• Experience in data modeling and distributed computing architectures.
• Hands-on experience with PySpark, Spark SQL, and Databricks notebooks/jobs.
• Experience orchestrating data pipelines with ADF, Airflow, or similar tools.
• Experience with both real-time and batch data processing.
• Experience building pipelines on Azure; AWS experience is a plus.
• Proficiency in SQL with window functions and performance optimization.
• Familiarity with DevOps tools, Git workflows, and CI/CD pipelines.
• Understanding of Scrum methodology and Scrum teams.
• Ability to apply Scrum practices in projects.
• Strong problem-solving and collaborative mindset.
• Experience with streaming technologies (Kafka, Flink, Kinesis).
• Ability to design and implement real-time data processing pipelines.

Responsibilities

• Design, develop, and maintain scalable data pipelines on Databricks and cloud platforms.
• Integrate data from databases, APIs, logs, streaming platforms, and external providers
• Develop data transformation routines to clean, normalize, and aggregate data
• Apply data processing techniques to handle complex datasets
• Contribute to frameworks and best practices for code development and deployment
• Implement data governance in alignment with company standards
• Partner with analytics and product leaders to design and operationalize pipelines
• Collaborate with infrastructure leaders to advance cloud-based data platforms
• Explore new tools and techniques leveraging Azure, Databricks, or related platforms
• Monitor data pipelines to detect and resolve issues promptly
• Develop monitoring tools, alerts, and automated error-handling mechanisms
• Analyse business requirements and identify data extraction requirements
• Attend requirement grooming and refinement sessions with users
• Develop and maintain ETL pipelines for ingestion, transformation, validation, and loading
• Optimise performance and batch scheduling
• Develop dashboards, reports, scorecards, and data visualizations
• Perform SIT, data validation, data profiling and confirm data accuracy
• Validate completeness and consistency of ETL Loads
• Support UAT and production implementation

Skills

Data modeling

Distributed computing

SQL

Data governance

Problem solving

Collaborative mindset

Scrum

Tools

Databricks

Azure

Airflow

Azure Data Factory

Spark SQL

Kafka

Git

CI/CD

Python

Job description

NCS Group is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms in Singapore. You will integrate diverse data sources, ensure data quality, and support analytics, reporting, and ML workloads while aligning with governance and reliability standards.

The role collaborates with analytics, product, and infrastructure teams to enhance the data platform, implement real-time and batch processing, and advance cloud-based capabilities using