Real-Time Data Engineer: Kafka, Spark, Databricks on AWS

Penbrothers; Mandaluyong; On-site; PHP 900,000 - 1,500,000; Full time

Job summary

Penbrothers is seeking a Data Engineer to design and operate high-throughput data pipelines in a cloud-native, real-time environment. You will work with Apache Kafka, Spark, Flink, RabbitMQ, Databricks, and Airflow to build scalable data platforms and APIs for internal use.

The role emphasizes software-like engineering practices, version control, CI/CD, testing, and collaboration with Product, BI, Engineering, and Infrastructure teams. Office in Mandaluyong with standard daytime hours.

Qualifications

• Advanced hands-on experience with streaming and real-time data technologies such as Apache Kafka, Spark (Core/Streaming), or RabbitMQ.
• Hands-on experience with Databricks and its ecosystem.
• Proven experience building distributed data pipelines using Spark and Apache Airflow.
• Strong programming in Python or Java with solid software engineering fundamentals.
• Advanced SQL skills and experience with MySQL, PostgreSQL, MongoDB, Elasticsearch.
• Experience architecting solutions within the AWS cloud ecosystem (e.g., EMR, MSK, Glue, S3).

Responsibilities

• Design, build, and maintain high-throughput, scalable data pipelines within a Databricks ecosystem.
• Develop, optimize, and maintain real-time streaming and batch data processing workflows.
• Ensure data quality, reliability, and observability across live data streams.
• Build high-performance data APIs and services for internal/external use.
• Troubleshoot production infrastructure and streaming pipeline issues.
• Collaborate with Product, BI, Engineering, and Infrastructure teams.
• Participate in code reviews and Agile ceremonies.

Skills

Distributed systems

Real-time streaming

Apache Kafka

Spark

Flink

RabbitMQ

Databricks

Airflow

Python

Java

APIs

SQL

AWS

Tools

Databricks

Airflow

Kafka

Spark

RabbitMQ

Job description

Penbrothers is seeking a Data Engineer to design and operate high-throughput data pipelines in a cloud-native, real-time environment. You will work with Apache Kafka, Spark, Flink, RabbitMQ, Databricks, and Airflow to build scalable data platforms and APIs for internal use.

The role emphasizes software-like engineering practices, version control, CI/CD, testing, and collaboration with Product, BI, Engineering, and Infrastructure teams. Office in Mandaluyong with standard daytime hours.