AWS Data Engineer: Scale Data Lakes with PySpark & ETL

INTELLECT MINDS PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

INTELLECT MINDS PTE. LTD. in Singapore seeks a data engineering professional to design, develop and maintain scalable data pipelines using PySpark on AWS, focusing on data lakes and lakehouse architectures.

The role requires strong engineering practices, Terraform IaC, CI/CD, and collaboration with stakeholders to deliver reliable data solutions. You will work with Python, SQL, AWS Glue, Step Functions, Lambda, and DevOps standards to ensure performance, security and cost efficiency.

Qualifications

• Strong hands-on experience with PySpark and Python for data engineering.
• Experience developing ETL pipelines using AWS Glue.
• Proficiency with AWS Step Functions for workflow orchestration.
• Experience building serverless solutions using AWS Lambda.
• Hands-on experience with Terraform for Infrastructure as Code (IaC).
• Experience implementing CI/CD pipelines using GitLab, GitHub Actions, Jenkins, or similar.
• Good understanding of AWS services, data lakes, IAM, S3, CloudWatch, and monitoring.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark on AWS.
• Build and orchestrate ETL/ELT workflows using AWS Glue and AWS Step Functions.
• Develop serverless applications and automation using AWS Lambda.
• Write clean, efficient, and maintainable Python/PySpark code following engineering best practices.
• Provision and manage cloud infrastructure using Terraform (Infrastructure as Code).
• Implement and maintain CI/CD pipelines to automate code deployment, testing, and infrastructure changes.
• Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and cost efficiency.
• Collaborate with business stakeholders to deliver data solutions.
• Follow DevOps, security, and coding standards throughout the engagement.

Skills

PySpark

Python

AWS

SQL

Data pipelines

Airflow

Terraform

CI/CD

Tools

AWS Glue

AWS Step Functions

AWS Lambda

Terraform (IaC)

GitLab / GitHub Actions / Jenkins

Job description

INTELLECT MINDS PTE. LTD. in Singapore seeks a data engineering professional to design, develop and maintain scalable data pipelines using PySpark on AWS, focusing on data lakes and lakehouse architectures.

The role requires strong engineering practices, Terraform IaC, CI/CD, and collaboration with stakeholders to deliver reliable data solutions. You will work with Python, SQL, AWS Glue, Step Functions, Lambda, and DevOps standards to ensure performance, security and cost efficiency.