Senior Data Engineer: Real-Time Spark, Kafka & NiFi

JobCubby; Hinoba-an; On-site; PHP 1,500,000 - 2,300,000; Full time

Job summary

Mastercard Philippines seeks aSenior Data Engineer with deep expertise in Spark, Kafka, and NiFi to design and maintain large-scale data platforms. You will work across teams to validate, optimize and productionize data pipelines.

The role emphasizes distributed systems, streaming architectures, and cloud-native data processing, with a focus on reliability, scalability, and operational excellence in production environments.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications.
• Strong expertise in Apache Spark (Batch and Structured Streaming).
• Strong experience with Apache Kafka and event-driven architectures.
• Experience developing, testing and supporting Apache NiFi data flows.
• Proficiency in Scala, pyspark or Python.
• Strong SQL and data modeling skills.
• Experience with distributed storage systems like Ozone, Ceph, HDFS, or cloud stores.
• Hands-on with Linux, Git, shell scripting, and CI/CD pipelines.
• Strong analytical and communication skills.
• Experience operating production-grade distributed systems in cloud/hybrid environments.

Responsibilities

• Develop, test, validate and maintain scalable batch and real-time data processing apps using Spark, Kafka, NiFi.
• Build validation suites for high-performance, fault-tolerant distributed systems.
• Develop reusable frameworks, libraries and shared components to accelerate engineering productivity.
• Translate business and technical requirements into scalable software solutions.
• Contribute to architecture discussions, designs and standards.
• Create validation suites for Spark batch/streaming apps to process high-volume data.
• Validate and support NiFi data flows.
• Build reliable data pipelines for real-time processing.
• Implement data replay, checkpoint management and failure handling.
• Analyze bottlenecks and optimize performance and resource use.
• Improve scalability and availability of distributed apps.
• Develop deployment automation and tooling for reliability.
• Monitor production and address operational issues.
• Participate in code reviews and documentation.

Skills

Spark (Batch & Streaming)

Kafka

NiFi

Scala/PySpark/Python

SQL & data modeling

Linux & CI/CD

Distributed systems

Education

Bachelor's degree in CS/Engineering

Tools

Apache Ozone

Ceph

HDFS

Docker

Kubernetes

Git

Job description

Mastercard Philippines seeks aSenior Data Engineer with deep expertise in Spark, Kafka, and NiFi to design and maintain large-scale data platforms. You will work across teams to validate, optimize and productionize data pipelines.

The role emphasizes distributed systems, streaming architectures, and cloud-native data processing, with a focus on reliability, scalability, and operational excellence in production environments.