Data Quality Engineer: Spark/Hive DataOps & Pipelines

Shopee

Philippines

On-site

PHP 900,000 - 1,300,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Shopee in the Philippines seeks a data engineer to design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables. You will develop and operate production batch ETL pipelines on Spark and Hive, and plan migrations with robust partition strategies.

You will also build cross-region pipelines for onboarding markets and business lines, plus create internal tools for data operations and data-quality checks, including AI-assisted

Qualifications

  • Master's degree in Computer Science, Software Engineering, AI from a reputable university.
  • Hands-on experience building and supporting production batch data pipelines with Spark/Spark SQL and Hive familiarity with HDFS and Hive Metastore.
  • Practical data-warehouse experience, including dimensional or SCD2 modeling, ETL design, partitioning, and backfill or recovery.
  • Strong fundamentals in data structures, algorithms, operating systems, and relational databases.
  • Strong SQL and proficiency in Python or Java able to write complex transformations involving nested JSON, window functions, and query tuning.
  • Able to diagnose Spark performance issues using the Spark UI and execution plans, including joins, shuffle, skew, and resource settings; working knowledge of Linux and Git.
  • Strong ownership and communication: can trace data issues end to end, validate a repair, and explain findings and data limitations to partner teams.

Responsibilities

  • Design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables.
  • Develop and operate production batch ETL pipelines on Spark and Hive, including onboarding, incremental loads, scheduled runs, and backfills.
  • Plan and execute data-store migrations, including Hive-to-ClickHouse migrations; design partition strategies and validate query behavior across regions.
  • Build reusable cross-region models and parameterized pipelines for onboarding markets, partners, and business lines.
  • Create lightweight internal tools for data operations, such as tooling, logging, monitoring, and interfaces; automate metadata discovery and data-quality checks, including AI-assisted workflows.
  • Maintain and optimize data ingestion pipelines for real-time updates, batch processing, and large-scale loads.
  • Develop and maintain a Metric Insight system with dashboards and AI-powered analytics.

Skills

Spark/Spark SQL
SQL
Python
Java
Data pipelines
Linux
Git

Education

Master's degree in Computer Science, Software Engineering, Artificial Intelligence

Tools

Hive
HDFS
Hive Metastore

Job description

Shopee in the Philippines seeks a data engineer to design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables. You will develop and operate production batch ETL pipelines on Spark and Hive, and plan migrations with robust partition strategies.

You will also build cross-region pipelines for onboarding markets and business lines, plus create internal tools for data operations and data-quality checks, including AI-assisted

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Quality Engineer, MPID Team
Data Quality Engineer, MPID Team

Shopee • Philippines

On-site
PHP 900,000 - 1,300,000
Data Engineer II: End-to-End Data Pipelines & Lakehouse
Data Engineer II: End-to-End Data Pipelines & Lakehouse

Digital Room (Philippines) , Inc. • Mandaluyong

On-site
PHP 900,000 - 1,400,000
Senior Data Engineer - End-to-End Pipelines on AWS
Senior Data Engineer - End-to-End Pipelines on AWS

Iscale Solutions • Makati

On-site
PHP 800,000 - 1,200,000
Senior Data Engineer End-to-End Pipelines On AWS
Senior Data Engineer End-to-End Pipelines On AWS

Iscale Solutions • Makati

On-site
PHP 800,000 - 1,200,000
Senior Data Engineer: AI-Driven Pipelines on AWS
Senior Data Engineer: AI-Driven Pipelines on AWS

Iscale Solutions • Makati

On-site
PHP 1,200,000 - 1,800,000
Databricks Data Engineer - Pipelines & Quality Lead
Databricks Data Engineer - Pipelines & Quality Lead

PERSOL • Santo Niño 1st

On-site
PHP 600,000 - 1,000,000
Big Data Backend Engineer Intern — Kubernetes & AI
Big Data Backend Engineer Intern — Kubernetes & AI

Shopee • Philippines

On-site
PHP 335,000 - 502,000
Data Engineer - Pipelines & Warehousing (Hybrid)
Data Engineer - Pipelines & Warehousing (Hybrid)

HRTx Inc. • Mandaluyong

Hybrid
PHP 600,000 - 1,200,000
Data Engineer: Build Scalable Pipelines & Data Lakes
Data Engineer: Build Scalable Pipelines & Data Lakes

Prudential • Taguig

On-site
PHP 1,200,000 - 2,400,000
Data Engineer | Hybrid/Remote, Growth & Impact
Data Engineer | Hybrid/Remote, Growth & Impact

Manatal • Philippines

Hybrid
PHP 1,004,000 - 1,674,000
Healthcare on day one + dependents
Bonuses
Paid time-off with cash conversion
+1
Top locations