Senior Data Platform Engineer

Tookitaki; Manila; On-site; PHP 1,500,000 - 2,100,000; Full time

Job summary

Tookitaki is seeking a Senior Data Platform Engineer to build and operate large-scale data platforms spanning on‑prem Cloudera and cloud-native AWS/Kubernetes environments. You will design batch and streaming pipelines with Spark, Flink, Hive, and associated tech, while handling storage, partitioning, and data models.

You will troubleshoot, optimize, and automate workloads, contributing to reliable, observable cloud-native data systems and migrations from VM/physical setups to AWS/Kubernetes.

Qualifications

• Extensive hands-on experience with production Cloudera platforms (on‑prem and cloud).
• Deep knowledge of Hadoop ecosystem: HDFS, Hive, YARN, Spark, and data pipelines.
• Proven ability to design, optimize, and operate batch and streaming workloads (Spark/Flink).
• Experience with Parquet/Avro/JSON formats, and storage/layout optimization.
• Strong automation and scripting skills (Python/Bash) and IaC tooling (Terraform/Ansible).

Responsibilities

• Build and operate large-scale data platforms across Cloudera on‑prem and cloud environments.
• Design batch and streaming pipelines using Spark, Flink, Hive; transform and optimize datasets.
• Administer clusters, perform upgrades, tuning, security, and backup/recovery tasks.
• Troubleshoot failed jobs, data skew, and resource contention; improve recovery and observability.
• Migrate workloads to AWS/Kubernetes, and implement GitOps-based delivery and automation.

Skills

Cloudera/Hadoop stack

AWS cloud

Kubernetes

Spark

Flink

Hive

HDFS

Data pipeline design

Troubleshooting production systems

Python/Bash automation

Terraform/Ansible

Observability/Monitoring

Security and data integrity basics

Tools

Cloudera Enterprise

AWS EMR

Kubernetes operators

Helm

Terraform

Ansible

CloudWatch

S3

IAM

Job description

Senior Data Platform Engineer — Cloudera, AWS & Kubernetes The mission We are looking for a battle-tested Data Platform Engineer who can build, operate, troubleshoot, and evolve large-scale data platforms across on-premises Cloudera environments and cloud-native AWS/Kubernetes architectures.

This is not a dashboard or SQL-only role. You will work where distributed compute, storage, networking, Kubernetes, and production data pipelines meet. You must be comfortable tracing a failed workload from the application layer through Spark or Flink, Kubernetes operators, HDFS/Hive, infrastructure, and AWS services.

The goal is to help us evolve safely from on-premises, VM, and EC2-based platforms into resilient, observable, cloud-native data systems.

What you'll do

• Build and operate production data platforms across Cloudera on premises, Cloudera cloud environments, AWS EMR, and Kubernetes.
• Design batch and streaming pipelines using Spark, Flink, Hive, and related technologies.
• Transform large datasets through filtering, sorting, joining, aggregation, partitioning, enrichment, and restructuring.
• Work with Parquet, Avro, JSON, CSV, and other delimited or semi-structured formats.
• Design storage, partitioning, compression, retention, and lifecycle strategies across HDFS, Hive, and object storage.
• Design and maintain Hive schemas, tables, partitions, metadata, and data models.
• Administer Cloudera clusters, including installation, upgrades, configuration, scaling, patching, security, backup, and recovery.
• Troubleshoot unhealthy services, failed jobs, resource contention, data skew, small-file problems, metadata issues, and storage bottlenecks.
• Tune Spark and Flink workloads for memory, CPU, parallelism, shuffle behavior, checkpointing, and recovery.
• Operate AWS services such as EMR, S3, IAM, EC2, EKS, CloudWatch, KMS, and supporting networking services.
• Deploy and operate data workloads on Kubernetes using operators, Helm, custom resources, and GitOps-based delivery.
• Help migrate workloads from physical or virtual machines and EC2 into AWS and Kubernetes-based platforms.
• Separate compute from storage where appropriate while accounting for performance, resilience, security, and cost.
• Build monitoring, alerting, capacity management, and operational runbooks for critical data services.
• Automate platform provisioning and configuration using Terraform, Ansible, scripting, and CI/CD.
• Support production incidents involving failed pipelines, delayed data, cluster degradation, storage pressure, or infrastructure failure.
• Work with data engineering, infrastructure, security, and application teams to resolve problems across ownership boundaries.

What we're looking for

• Strong hands-on experience administering Cloudera platforms in production.
• Experience with both on-premises Cloudera and cloud-based Cloudera deployments.
• Deep working knowledge of Hadoop, HDFS, Hive, YARN, Spark, and the wider distributed-data ecosystem.
• Experience building or operating production workloads using Apache Flink.
• Strong understanding of distributed data processing, including partitioning, shuffling, serialization, checkpointing, and failure recovery.
• Experience transforming large datasets using joins, aggregations, filtering, sorting, and schema evolution.
• Practical knowledge of Parquet, Avro, JSON, CSV, compression formats, and serialization tradeoffs.
• Experience designing data layouts for query performance, ingestion throughput, retention, and cost.
• Strong AWS experience, particularly with EMR, S3, EC2, EKS, IAM, CloudWatch, and KMS.
• Strong Kubernetes experience, including operators, controllers, Helm, scheduling, storage, networking, and workload troubleshooting.
• Experience migrating data platforms from on-premises or VM-based environments into AWS and Kubernetes.
• Ability to troubleshoot Linux, JVM, networking, storage, DNS, certificates, and resource-management issues.
• Experience with observability platforms and the ability to correlate infrastructure symptoms with data-pipeline failures.
• Ability to automate operational work using Python, Bash, Terraform, Ansible, or equivalent tools.
• Strong judgment around production changes, data integrity, access control, rollback, and recovery.

Production scenarios you should be able to handle

• A Spark job that ran in 40 minutes yesterday now takes four hours.
• A join creates severe data skew and repeatedly exhausts executor memory.
• HDFS is approaching capacity while NameNode health is degrading.
• Hive queries return incomplete results because partitions or metadata are inconsistent.
• A Flink job repeated failures after checkpoint recovery.
• An EMR workload is reliable but significantly more expensive than expected.
• A Kubernetes operator reports success while the underlying data workload is unhealthy.
• A migrated workload behaves differently on S3 than it did on HDFS.
• A certificate, Kerberos, IAM, DNS, or network problem presents as an application failure.
• A critical pipeline misses its SLA and ownership is unclear across platform and data teams.

Certifications Relevant certifications are useful, particularly:

• Cloudera Certified Professional: Data Engineer
• Cloudera Administrator certification or equivalent production experience
• AWS data, analytics, or architecture certifications
• Kubernetes certifications such as CKA or CKAD

What success looks like

• Data pipelines meet their reliability and processing-time objectives.
• Platform failures are detected before downstream consumers report them.
• Incidents move quickly from symptoms to an evidence-backed root cause.
• Cloudera, AWS, and Kubernetes environments are operated through repeatable automation.
• Migrations preserve data correctness while improving scalability and operability.
• Storage and compute designs balance performance, resilience, and cost.
• Data engineers can ship workloads without becoming accidental platform administrators.
• Operational knowledge becomes monitoring, automation, and runbooks—not tribal memory.

The person we want You understand that a data platform is a distributed production system, not a collection of product names. You can move from a Hive execution plan to Spark executor logs, Kubernetes events, HDFS health, S3 behavior, IAM permissions, and network telemetry without losing the thread.

You know the architectural differences between on-premises Hadoop and cloud- native data platforms, including where a lift-and-shift approach will fail. We need someone who can enter a degraded platform, establish the facts, protect data integrity, restore service, explain the failure chain, and make the system harder to break next time.