Senior Data Engineer

RAPSYS TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A technology solutions provider in Singapore is seeking a Data Engineer to support data requirements and develop E2E Data Pipelines. The ideal candidate has a Bachelor's in Computer Science and at least 4 years of experience in data engineering, with skills in Python, SQL, and cloud technologies like Azure and AWS. Join a collaborative team to analyze data patterns, automate processes, and enhance data visualization solutions.

Qualifications

• Minimum of 4 years' experience in Data Engineering fields.
• Experience working in a Singapore public sector or consulting environment is a plus.
• Basic knowledge in Data Lake/Data Warehousing/Big Data tools.

Responsibilities

• Support data requirements including reports and dashboards.
• Design and develop the E2E Data Pipeline.
• Develop data pipeline automation using Azure, AWS, Databricks.
• Develop ETL using Spark processing.
• Generate data visualization using Power BI or Tableau.

Skills

Python

SQL

Spark

Cloud Architect

Data Visualization

DAX programming

API

AWS

DAX programming

Data Visualization

Education

Bachelor's degree in Computer Science or related fields

Tools

Databricks

Azure

AWS

Power BI

Tableau

Job description

Job Responsibilities

• Work across workstreams to support data requirements including reports and dashboards
• Analyze and perform data profiling to understand data patterns and discrepancies following Data Quality and Data Management processes
• Understand and follow best practices to design and develop the E2E Data Pipeline: data transformation, ingestion, processing, and surfacing of data for large-scale applications
• Develop data pipeline automation using Azure, AWS data platform and technologies stack, Databricks, Data Factory
• Understand business requirements to translate them into technical requirements that the system analysts and other technical team members can drive into the project design and delivery
• Analyze source data and perform data ingestion in both batch and real-time patterns via various methods; for example, file transfer, API, Data Streaming using Kafka and Spark Streaming
• Analyze and understand data processing and standardization requirements, develop ETL using Spark processing to transform data
• Understand data/reports and dashboards requirements, develop data export, data API, or data visualization using Power BI, Tableau, or other visualization tools

Qualification

We are looking for experience and qualifications in the following:

• Bachelor’s degree in Computer Science, Computer Engineer, IT, or related fields
• Minimum of 4 years’ experience in Data Engineering fields
• Data Engineering skills: Python, SQL, Spark, Cloud Architect, Data & Solution Architect, API, Databricks, Azure, AWS
• Data Visualization skills: Power BI (or other visualization tools), DAX programming, API, Data Model, SQL, Story Telling and wireframe design
• Business Analyst skills: business knowledge, data profiling, basic data model design, data analysis, requirement analysis, SQL programing
• Basic knowledge in Data Lake/Data Warehousing/ Big data tools, Apache Spark, RDBMS and NoSQL, Knowledge Graph
• Experience working in a Singapore public sector, client-facing/consulting environment is a plus
• Team player, analytical and problem-solving skills

Mandatory Skills: PySpark, Databricks