Data Quality Engineer, MPID Team

Shopee

Philippines

On-site

PHP 900,000 - 1,300,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Shopee in the Philippines seeks a data engineer to design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables. You will develop and operate production batch ETL pipelines on Spark and Hive, and plan migrations with robust partition strategies.

You will also build cross-region pipelines for onboarding markets and business lines, plus create internal tools for data operations and data-quality checks, including AI-assisted

Qualifications

  • Master's degree in Computer Science, Software Engineering, AI from a reputable university.
  • Hands-on experience building and supporting production batch data pipelines with Spark/Spark SQL and Hive familiarity with HDFS and Hive Metastore.
  • Practical data-warehouse experience, including dimensional or SCD2 modeling, ETL design, partitioning, and backfill or recovery.
  • Strong fundamentals in data structures, algorithms, operating systems, and relational databases.
  • Strong SQL and proficiency in Python or Java able to write complex transformations involving nested JSON, window functions, and query tuning.
  • Able to diagnose Spark performance issues using the Spark UI and execution plans, including joins, shuffle, skew, and resource settings; working knowledge of Linux and Git.
  • Strong ownership and communication: can trace data issues end to end, validate a repair, and explain findings and data limitations to partner teams.

Responsibilities

  • Design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables.
  • Develop and operate production batch ETL pipelines on Spark and Hive, including onboarding, incremental loads, scheduled runs, and backfills.
  • Plan and execute data-store migrations, including Hive-to-ClickHouse migrations; design partition strategies and validate query behavior across regions.
  • Build reusable cross-region models and parameterized pipelines for onboarding markets, partners, and business lines.
  • Create lightweight internal tools for data operations, such as tooling, logging, monitoring, and interfaces; automate metadata discovery and data-quality checks, including AI-assisted workflows.
  • Maintain and optimize data ingestion pipelines for real-time updates, batch processing, and large-scale loads.
  • Develop and maintain a Metric Insight system with dashboards and AI-powered analytics.

Skills

Spark/Spark SQL
SQL
Python
Java
Data pipelines
Linux
Git

Education

Master's degree in Computer Science, Software Engineering, Artificial Intelligence

Tools

Hive
HDFS
Hive Metastore

Job description

Job Description:
  • Design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables. Define each table's grain, keys, business-date and partition logic, metric definitions, dependencies, and backfill and release procedures.
  • Develop and operate production batch ETL pipelines on Spark and Hive, including partner and source onboarding, incremental loads, scheduled runs, and historical backfills. Use staging and checkpoint patterns where needed so retries do not corrupt published data or SCD2 history.
  • Plan and execute data-store or platform migrations, including Hive-to-ClickHouse migrations where required. Design partition strategies, write migration scripts, reconcile source and target data, and validate query behavior across regions.
  • Build reusable cross-region models and parameterized pipelines for onboarding markets, partners, and business lines. Standardize schemas and metric logic to reduce one-off implementations.
  • Build and maintain lightweight internal tools for data operations, such as task scaffolding, logging and monitoring, and query or configuration interfaces. Automate repetitive metadata discovery and data-quality checks, including through AI-agent or LLM-assisted workflows where appropriate.
  • Maintain and optimize data ingestion pipelines by designing solutions for real-time updates, batch processing, and initial full loads to meet different data-source and business requirements.
  • Develop and maintain a Metric Insight system that meets business and BI needs by standardizing metrics, building dashboards, and integrating AI-powered analytics tools to provide a one-stop data analysis platform.

Requirements:
  • Master's degree in Computer Science, Software Engineering, Artificial Intelligence from a reputable university.
  • Hands-on experience building and supporting production batch data pipelines with Spark/Spark SQL and Hive familiarity with HDFS and Hive Metastore.
  • Practical data-warehouse experience, including dimensional or SCD2 modeling, ETL design, partitioning, and backfill or recovery.
  • Strong fundamentals in data structures, algorithms, operating systems, and relational databases.
  • Strong SQL and proficiency in Python or Java able to write complex transformations involving nested JSON, window functions, and query tuning.
  • Able to diagnose Spark performance issues using the Spark UI and execution plans, including joins, shuffle, skew, and resource settings working knowledge of Linux and Git.
  • Strong ownership and communication: can trace data issues end to end, validate a repair, and explain findings and data limitations to partner teams.

Kindly note that you can only be considered for one role at a time with any of the companies within our Group. If you have applied for other jobs with the Group, you will be considered for roles in the order of your application

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

EiVA HR Solutions • Metro Manila

On-site
PHP 900,000 - 1,500,000
Senior Data Engineer (Remote)
Senior Data Engineer (Remote)

Iscale Solutions • Makati

On-site
PHP 1,200,000 - 1,800,000
Data Quality Engineer: Spark/Hive DataOps & Pipelines
Data Quality Engineer: Spark/Hive DataOps & Pipelines

Shopee • Philippines

On-site
PHP 900,000 - 1,300,000
Senior Data Engineer
Senior Data Engineer

PRO SOURCE GLOBAL BPO INC. • Makati

On-site
PHP 900,000 - 1,800,000
Data Engineer - Level 2
Data Engineer - Level 2

Digital Room (Philippines) , Inc. • Mandaluyong

On-site
PHP 900,000 - 1,400,000
Data Engineer
Data Engineer

Viventis Search Asia • Quezon City

On-site
PHP 900,000 - 1,500,000
Data Engineer - MRF05447
Data Engineer - MRF05447

Hunter's Hub Inc. • Taguig

On-site
PHP 669,600 - 892,800
Data Engineer
Data Engineer

Bloomberry Resorts and Hotels Inc. • Parañaque

On-site
PHP 900,000 - 1,500,000
Data Engineer
Data Engineer

Triseed Software Inc. • Taguig

Hybrid
PHP 900,000 - 1,400,000
Data Tech Lead
Data Tech Lead

WeSupport Incorporated • Taguig

Hybrid
PHP 1,200,000 - 2,400,000
Top locations