Senior Data Engineer – Real-Time Analytics on AWS

Coins.ph; Malaysia; On-site; MYR 368,000 - 613,000; Full time

Job summary

Coins.ph is seeking a senior Data Engineer to design and maintain scalable data processing systems on the Databricks platform. You will collaborate with analysts, software engineers, and stakeholders to build robust ETL pipelines and real-time data processing with Spark Streaming and Kafka.

You will leverage AWS services (EC2, S3, Glue) to deploy data infrastructure and enforce strong data quality, security, and governance practices across the organization.

Qualifications

• Bachelor’s or Master’s degree in Computer Science or a related field.
• Minimum of 5 years of hands-on experience as a Data Engineer, designing and implementing data processing systems.
• Good understanding of Databricks platform and Delta Lake.
• Familiar with data job scheduler tools such as Dagster.
• Proficiency in one or more languages such as Scala, Java, or Python.
• Deep expertise in big data technologies including Apache Spark for ETL processing and optimization.
• Proficient in BI tools such as Metabase for data visualization and analysis.
• Advanced understanding of data modeling, data quality, and data governance.
• Excellent communication and collaboration with diverse stakeholders.
• Extensive AWS operational management experience and cloud deployment.
• Strong knowledge of AWS services like EC2, S3, Glue, EMR for scalable data solutions.
• Proficiency in AWS security practices and access controls.
• Hands-on experience with Terraform for IaC and automation.

Responsibilities

• Design, develop, and maintain scalable data processing systems with focus on code quality and performance.
• Collaborate with data analysts, software developers, and business stakeholders to understand data needs and architecture.
• Develop and maintain ETL pipelines into our data warehouse on Databricks.
• Lead real-time data processing using Spark Streaming and Kafka.
• Establish data quality checks to ensure accuracy and consistency of data assets.
• Troubleshoot complex data processing issues with cross-functional teams.
• Monitor and optimize data processing systems for performance and reliability using AWS.
• Use AWS services (EC2, S3, Glue, Databricks) to deploy and manage cloud data infrastructure.
• Implement security measures and access controls to protect sensitive data in AWS.
• Stay updated with AWS tools and best practices to continually improve data capabilities.

Skills

Spark

Python

Java

Scala

Metabase

Big data

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Databricks

Delta Lake

Dagster

Terraform

AWS

EC2

S3

Glue

Kafka

Job description

Coins.ph is seeking a senior Data Engineer to design and maintain scalable data processing systems on the Databricks platform. You will collaborate with analysts, software engineers, and stakeholders to build robust ETL pipelines and real-time data processing with Spark Streaming and Kafka.

You will leverage AWS services (EC2, S3, Glue) to deploy data infrastructure and enforce strong data quality, security, and governance practices across the organization.