Senior Data Engineer - Cloud Data Pipelines & Spark

Experian; Petaling Jaya; Hybrid; MYR 100,000 - 201,000; Full time

Benefits offered by this job

Hybrid work arrangement: home and offi

Inclusive paid leave

People-first and inclusive workplace

Paid Internship

Job summary

Experian is seeking a talented data engineer to join the Ascend Sandbox team, Experian's cloud-based analytical platform. You will build, maintain and scale data pipelines serving internal teams and external clients across the APAC region.

Join a hybrid work environment to work on scalable ETL/ELT pipelines, data lakehouse practices and production-grade data solutions using Python, PySpark and AWS services.

Qualifications

• Degree in Computer Science, Data Engineering or related field.
• At least 6 years of experience in data engineering or data-intensive systems.
• Strong hands-on experience with Python and PySpark for large-scale processing.
• Strong SQL skills and data modelling for analytics.
• Experience with AWS data services: S3, Glue, Athena, Redshift, EMR.

Responsibilities

• Design, build, test, deploy and operate scalable data pipelines.
• Develop Python, PySpark and SQL solutions for large datasets.
• Design data models and reusable datasets for analytics and data science.
• Engineer and optimise AWS-based data solutions (S3, Glue, Athena, Redshift, EMR).
• Apply data lake/lakehouse practices including partitioning and formats (Parquet).
• Implement ETL/ELT patterns with incremental loads, retries and backfills.
• Implement workflow orchestration using Airflow/MWAA or Step Functions.
• Build automated data quality checks and monitoring for pipeline health.
• Collaborate with clients, data scientists and engineers to translate requirements.
• Ensure code quality, security and CI/CD in deployments.

Skills

Python

PySpark

SQL

ETL/ELT design

Data modelling

Spark performance engineering

Problem solving

Communication

Education

Degree in Computer Science or equivalent

Tools

Amazon S3

AWS Glue

Amazon Athena

Amazon Redshift

Amazon EMR

Airflow/MWAA

Terraform

AWS Step Functions

Jira

Bitbucket

Job description

Experian is seeking a talented data engineer to join the Ascend Sandbox team, Experian's cloud-based analytical platform. You will build, maintain and scale data pipelines serving internal teams and external clients across the APAC region.

Join a hybrid work environment to work on scalable ETL/ELT pipelines, data lakehouse practices and production-grade data solutions using Python, PySpark and AWS services.