Senior Data Engineer

UpGrowth HR Sdn. Bhd.; Kuala Lumpur; On-site; MYR 90,000 - 130,000; Full time

Job summary

Upgrowth HR. SDN BHD is seeking a skilled Data Engineer to design, build, and maintain scalable ETL/ELT pipelines for automated data ingestion, transformation, and validation in a growing data practice.

You will centralize data storage, start with lightweight solutions and scale toward cloud-based infrastructure; collaborate with analysts and business teams to model data, implement quality checks, and refactor workflows into production-ready codebases while promoting best practices in version

Qualifications

• Proficient in Python data processing (pandas, PySpark preferred).
• Strong SQL skills and relational database experience.
• Experience building ETL/ELT pipelines and workflow orchestration (Airflow/Prefect/Luigi).
• Data modeling, schema design, and data governance fundamentals.
• Ability to work with stakeholders and translate reporting needs into data solutions.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines for automated data ingestion, transformation, and validation.
• Establish a centralized data storage system, starting with lightweight solutions and scaling toward cloud-based infrastructure.
• Collaborate with analysts and business teams to translate reporting needs into scalable, reusable data models.
• Implement data quality checks, logging, and monitoring to ensure reliability and transparency of data flows.
• Refactor existing Python/Jupyter workflows into structured, production-ready codebases.
• Evaluate and recommend tools for data warehousing, orchestration, and cloud migration.
• Define and promote best practices in version control, documentation, and reproducibility.

Skills

Python

SQL

Data modeling

Data governance

Problem solving

Tools

Airflow

Prefect

Luigi

Snowflake

BigQuery

Redshift

Databricks

Docker

Power BI

Tableau

AWS

GCP

Azure

CI/CD

Job description

JOB RESPONSIBILITIES:

• Design, build, and maintain ETL/ELT pipelines for automated data ingestion, transformation, and validation.

• Establish a centralized data storage system, starting with lightweight solutions and scaling toward cloud-based infrastructure.

• Collaborate with analysts and business teams to translate reporting needs into scalable, reusable data models.

• Implement data quality checks, logging, and monitoring to ensure reliability and transparency of data flows.

• Refactor existing Python/Jupyter workflows into structured, production-ready codebases.

• Evaluate and recommend tools for data warehousing, orchestration, and cloud migration.

• Define and promote best practices in version control, documentation, and reproducibility.

JOB REQUIREMENTS:

• Strong proficiency in Python (pandas, PySpark, or similar libraries for data processing).

• Solid understanding of SQL and experience with relational databases.

• Hands-on experience designing or maintaining ETL pipelines and workflow orchestration tools (e.g., Airflow, Prefect, Luigi).

• Good grasp of data modeling, schema design, and data governance principles.

• Experience handling large datasets and optimizing data performance.

• Comfortable working in environments with minimal existing infrastructure and building systems from scratch.

• Strong analytical thinking, problem-solving skills, and ability to work with both technical and non-technical stakeholders.

NICE TO HAVE:

• Experience with cloud platforms (AWS, GCP, or Azure) and cloud-native data services.

• Knowledge of data warehousing technologies (Snowflake, BigQuery, Redshift, or Databricks).

• Familiarity with CI/CD pipelines, Docker, or containerized workflows.

• Exposure to business intelligence tools (Power BI, Tableau, or similar).