Senior Data Engineer

5G-Starlink Pte.; Malacca City; On-site; MYR 150,000 - 210,000; Full time

Job summary

Deriv seeks a Senior Data Engineer to define future data models and pipelines. You will leverage GCP, Airflow, Python, Docker, and PostgreSQL to deliver dependable BI solutions. Build, test, and maintain ETL/ELT architectures and ensure trustworthy data warehouses for analytics.

You will extract from diverse sources, maintain data integrity, and enable data-driven business decisions through scalable pipelines and modern storage formats.

Qualifications

• Minimum 5 years in data engineering.
• Experience with Kimball star schema, Anchor modelling or Data Vault.
• Proficient in Python for data processing.
• Strong SQL and NoSQL skills, preferably PostgreSQL.
• Familiarity with data warehouses and columnar storage.
• Hands-on with GCP services and ETL/ELT pipelines.

Responsibilities

• Design, build and maintain data pipelines and data warehouse.
• Ensure data integrity and security across sources.
• Develop scalable ETL/ELT architectures using Airflow.
• Translate data into actionable business insights for analyses.
• Collaborate with data analysts and data scientists.
• Stay updated on product releases impacting data models.

Skills

Data modelling

Python

SQL/NoSQL

ETL/ELT

Airflow

GCP

Data warehousing

Communication

English proficiency

Education

5+ years experience in data engineering

Tools

PostgreSQL

Airflow

Docker

BigQuery

Redshift

Parquet

Pg_basebackup

WAL archival

Job description

We are the Business Intelligence team. We lead the organisation in cultivating a data-driven culture as our company moves toward the future. We collect meaningful data and analytics for us to deeply understand our consumers and build more valuable products and services. What we do is incredibly important in driving smart marketing decisions, optimising our business, and increasing profitability.

Your role

At Deriv, we are looking for a detail-oriented Senior Data Engineer with the capability to define the data pipeline of our future data models. You will leverage technologies such as Google Cloud Platform, Airflow, Python, Docker, and PostgreSQL to help provide the company with dependable business intelligence solutions. As part of this role, you will develop, test, and maintain architectures for data processing and build Extract, Transform, and Load (ETL) pipelines. You will be a key contributor to making our data warehouse trustworthy by ensuring data accuracy.

• Ensure data integrity while extracting data from in-house and third-party complex sources and manage its systematic storage. Responsible for data security, accuracy, and accessibility
• Provide tangible business solutions and decisions using your expertise in the data engineering domain
• Design and build high-performance, secure, and scalable company data warehouse and pipeline to support data science projects following best practices
• Debug and resolve complex issues, and recommend improvements to ensure a well-functioning ETL pipelined architecture
• Transform raw data into easy-to-use tables for the Data Analysts
• Keep up-to-date on company products and new releases to efficiently plan changes in our data warehouse or pipelines
  

What you have

• A minimum of 5 years of experience in the data engineering field
• Expertise in data modelling techniques such as Kimball star schema, Anchor modelling, and Data vault
• Competence in object-oriented or object function scripting languages such as Python
• Proficiency in relational SQL and NoSQL databases, preferably with PostgreSQL, PITR, Pg_basebackup, WAL archival, and Replication
• Familiarity with column-oriented storage or data warehouses such as parquet, Redshift, and Bigquery
• In-depth skills in developing and maintaining ETL/ELT data pipelines and workflow management tools such as Airflow
• Hands-on experience with Google Cloud Platform (GCP) services such as BigQuery, scheduled queries, Cloud Storage and Cloud Functions
• Familiar with alerting and self-recovery methods concerning data accuracy
• Analytical skills with the ability to transform data into optimal business decisions
• Expertise in peer reviewing pipeline codes and suggesting improvements when required
• Experience in helping teams make informed business decisions with data
• Strong communication and presentation skills
• Fluency in spoken and written English

What's good to have

• A good background in cybersecurity and data protection
• Proficiency in using data pipeline and workflow management tools such as Luigi
• Exposure to maintaining and monitoring database health and resolving errors
• Experience in managing stakeholders expectations and technical requirement gathering
• Familiarity with container technologies such as Docker