Senior Data Engineer

RAPSYS TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 70,000 - 90,000; Full time

Job summary

A data solutions firm in Singapore is seeking a Data Engineer to design and develop end-to-end data pipelines while ensuring data quality. The ideal candidate has a Bachelor's degree in Computer Science and a minimum of 4 years of experience in Data Engineering, with skills in Python, SQL, Databricks, and cloud platforms like Azure and AWS. Join a dynamic team to work on innovative data solutions.

Qualifications

• Minimum of 4 years’ experience in Data Engineering fields.
• Experience working in a Singapore public sector, client-facing/consulting environment is a plus.
• Team player, analytical and problem-solving skills.

Responsibilities

• Work across workstreams to support data requirements including reports and dashboards.
• Analyze and perform data profiling to understand data patterns and discrepancies.
• Understand and follow best practices to design and develop the E2E Data Pipeline.
• Develop data pipeline automation using Azure and AWS.
• Analyze source data and perform data ingestion in various methods.

Skills

Python

SQL

Spark

Cloud Architect

Data Visualization

DAX programming

Business Analysis

Apache Spark

Education

Bachelor's degree in Computer Science, Computer Engineer, IT, or related fields

Tools

Databricks

Azure

AWS

Power BI

Tableau

Job description

Job Responsibilities

• Work across workstreams to support data requirements including reports and dashboards
• Analyze and perform data profiling to understand data patterns and discrepancies following Data Quality and Data Management processes
• Understand and follow best practices to design and develop the E2E Data Pipeline: data transformation, ingestion, processing, and surfacing of data for large-scale applications
• Develop data pipeline automation using Azure, AWS data platform and technologies stack, Databricks, Data Factory
• Understand business requirements to translate them into technical requirements that the system analysts and other technical team members can drive into the project design and delivery
• Analyze source data and perform data ingestion in both batch and real-time patterns via various methods; for example, file transfer, API, Data Streaming using Kafka and Spark Streaming
• Analyze and understand data processing and standardization requirements, develop ETL using Spark processing to transform data
• Understand data/reports and dashboards requirements, develop data export, data API, or data visualization using Power BI, Tableau, or other visualization tools

Qualifications

We are looking for experience and qualifications in the following:

• Bachelor’s degree in Computer Science, Computer Engineer, IT, or related fields
• Minimum of 4 years’ experience in Data Engineering fields
• Data Engineering skills: Python, SQL, Spark, Cloud Architect, Data & Solution Architect, API, Databricks, Azure, AWS
• Data Visualization skills: Power BI (or other visualization tools), DAX programming, API, Data Model, SQL, Story Telling and wireframe design
• Business Analyst skills: business knowledge, data profiling, basic data model design, data analysis, requirement analysis, SQL programing
• Basic knowledge in Data Lake/Data Warehousing/ Big data tools, Apache Spark, RDBMS and NoSQL, Knowledge Graph
• Experience working in a Singapore public sector, client-facing/consulting environment is a plus
• Team player, analytical and problem-solving skills

Mandatory Skills: PySpark, Databricks