ETL Developer 24 days ago

Digital Room; Mandaluyong; On-site; PHP 900,000 - 1,300,000; Full time

Job summary

Digital Room is seeking a Data Engineer 2 in Manila to own end-to-end data pipelines, from ingestion to publishing, with hands-on responsibility for orchestration, lineage, and dataset modeling. You will build robust batch, streaming, and CDC-enabled pipelines, optimize storage formats, and ensure data quality and lineage across platforms.

The role emphasizes design decisions beyond individual tasks, with a focus on scalable data architectures, reproducible data, and clear documentation.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or related field, or equivalent practical experience.
• At least 3 years building and operating production data pipelines (not solely analytics or reporting).
• Strong Python at production quality: modular code, error handling, logging, configuration, and tests.
• Advanced SQL, including window functions, multi-step CTE transformations, and set operations; read an execution plan and explain slow queries.
• Hands-on AWS data services: S3, Glue, Lambda, and Redshift, plus at least one of DMS, Kinesis, EMR, Step Functions, or Athena.
• Experience loading and querying non-transactional stores, including a data lake on object storage.
• Production experience with at least one orchestration framework, and understanding of workflow design, dependencies, idempotency, retries, and backfills.
• Data modeling ability: dimensional and/or wide-table designs, grain and key definition, and slowly changing dimensions.
• Data quality as routine practice: automated tests and reconciliation; CI-based deployment.
• Clear written and verbal communication.

Responsibilities

• Build and maintain batch, incremental, and near-real-time pipelines from various sources to data stores.
• Implement change data capture (CDC) on high-volume source tables, handling drift and out-of-order records.
• Write idempotent, restartable jobs ensuring consistent results on re-runs.
• Apply load patterns (full refresh, watermark-based incremental, merge/upsert, slowly changing dimensions).
• Write and tune complex SQL; read query plans and address performance issues.
• Apply business rules in a testable layer; design orchestration and observability.
• Model data with clear grain, keys, and measures; consider Parquet/ORC formats and storage layouts.
• Maintain dataset catalogs, perform impact analysis before schema changes, enforce data quality gates.
• Provide production support and on-call incident response; document datasets and pipelines clearly.
• Mentor Level 1 engineers; participate in design/code reviews to improve maintainability and cost.

Skills

Python production quality

Advanced SQL

AWS data services

Data modeling

Data quality practices

Git CI/CD

Communication skills

Education

Bachelor’s degree in Computer Science, Information Technology, or related field

Tools

dbt

Dagster

Prefect

Terraform

CloudFormation

CDK

Spark

Trino

Athena

OpenLineage/Marquez

Job description

Position Summary The Data Engineer 2 builds and operates the pipelines and datasets that the business runs on. This is a hands-on delivery role: the engineer owns pipelines end to end (ingestion, transformation, publishing, monitoring, and support) and is expected to contribute to design decisions above the level of an individual job, particularly in orchestration, data lineage, and how curated data is modeled and exposed to downstream consumers.

Beyond conventional ETL, this role builds the curated, well-described datasets that analytics tools and AI systems consume. A successful candidate can explain not only how to move data, but why a given storage format, partition layout, or table design is the right one, and what breaks downstream when it is wrong. Responsibilities

Build and maintain batch, incremental, and near-real-time pipelines that extract from transactional databases, SaaS and internal APIs, file feeds, and event streams, and load to databases, data lakes, and data warehouses. Implement change data capture (CDC) on high-volume source tables, handling schema drift, late-arriving and out-of-order records, soft deletes, and replays. Write idempotent, restartable jobs: re-running a pipeline for a given partition or window produces the same result without duplicating or losing rows.

Apply the correct load pattern for the volume and business need, whether full refresh, watermark-based incremental, merge/upsert, or slowly changing dimension (Type 1 and Type 2), and justify the choice. Write and tune complex SQL: window functions, multi-CTE transformations, set operations. Read the query plan and address the actual cause of slowness (scan volume, distribution skew, spill to disk, poor join order) rather than adding hardware.

Apply business rules and integration logic in a testable, documented layer rather than embedded in ad-hoc scripts. Designing the orchestration layer, task dependencies, and observability. Responsibilities include implementing backfill mechanisms, managing failure strategies, and creating thorough documentation.

Modeling data for consumption with clear definitions of grain, keys, and measures, alongside an understanding of columnar storage mechanics like Parquet and ORC. Responsibilities include applying physical design principles for columnar warehouses, managing file layouts in object storage, and working with non-transactional stores while evaluating and defending the appropriate storage solution for various workloads. Designing automated lineage capture at table and column levels, ensuring that documentation stays current with orchestration and transformation changes.

Responsibilities include maintaining dataset catalogs with essential details, performing impact analysis before schema changes, and implementing data quality gates to prevent erroneous data from being published. Additionally, focuses on supporting auditability and reproducibility while applying least-privilege access and data retention rules. Thorough testing before code promotion, including unit tests and validation against source counts, along with ensuring the target environment is ready for initial data loads.

Providing production support for pipelines, managing failures with a focus on root causes, participating in an on-call rotation for incidents, and working within a Git-based workflow with peer reviews. Additionally, comprehensive documentation is essential to ensure that datasets and pipelines can be operated by others without assistance. Translating business questions from various stakeholders into clear data requirements while suggesting alternative solutions when appropriate.

Responsibilities include participating in design and code reviews to address concerns regarding maintainability, cost, and data accuracy, as well as mentoring Level 1 engineers in SQL, pipeline patterns, and debugging techniques. Provide production support for the pipelines you own: triage and root-cause failures, communicate impact and ETA to affected consumers, then eliminate the class of failure rather than only the instance.

Required Qualifications

Bachelor’s degree in Computer Science, Information Technology, or a related field, or equivalent practical experience. At least 3 years building and operating production data pipelines (not solely analytics or reporting work). Strong Python at production quality: modular code, error handling, logging, configuration, and tests, well beyond one-off scripts.

Advanced SQL, including window functions, multi-step CTE transformations, and set operations; able to read an execution plan and explain why a query is slow and what to change. Hands-on AWS data services: S3, Glue, Lambda, and Redshift, plus at least one of DMS, Kinesis, EMR, Step Functions, or Athena. Demonstrable understanding of columnar storage and columnar analytical databases: file formats, encoding and compression, partitioning and pruning, and their effect on query cost.

Experience loading and querying non-transactional stores, including a data lake on object storage; able to explain how lake and warehouse workloads differ. Production experience with at least one orchestration framework, and the ability to reason about workflow design, dependencies, idempotency, retries, and backfills rather than only clicking through a UI. Data modeling ability: dimensional and/or wide-table designs, grain and key definition, and slowly changing dimensions.

Data quality as routine practice: automated tests and reconciliation as part of delivery, not an afterthought. Experience with Git branching, pull requests, and CI-based deployment. Clear written and verbal communication.

Able to document a dataset so that someone who did not build it can use it correctly, and able to explain a data issue to a non-technical stakeholder. Ability to analyze broad, ambiguous data needs and decompose them into deliverable work.

Preferred Qualifications

Open table formats (Apache Iceberg, Delta Lake, Hudi) and lakehouse architecture in production. A transformation framework with built-in testing, documentation, and lineage, such as dbt. Asset-based or flow-based orchestration (Dagster, Prefect) in addition to AWS-native workflow services.

Lineage and catalog tooling: AWS Glue Data Catalog, OpenLineage/Marquez, DataHub, OpenMetadata, Amundsen, Collibra, or Alation. Streaming and CDC at scale: Kinesis, Kafka/MSK, Debezium, or equivalent, including exactly-once and ordering concerns. Distributed query and processing engine tuning: Spark, Trino, or Athena.

Exposure to serving data to LLMs or AI agents, such as MCP servers, semantic layers, text-to-SQL, or retrieval and vector stores, plus awareness of the failure modes (ambiguous columns, unbounded scans, leaked sensitive fields, stale context). Infrastructure as code (Terraform, CloudFormation, or CDK) and containerized workloads. Cost awareness and optimization: attributing and reducing storage, scan, and compute cost.

Familiarity with the downstream BI layer and how modeling decisions constrain it. Domain exposure to e-commerce, print or custom manufacturing, or high-volume order data: orders, SKUs and product configuration, pricing, fulfillment, and marketing attribution.