Senior Data Engineer (AWS)

Intellect Minds Pte Ltd; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

Intellect Minds Pte Ltd seeks a data engineer to design and maintain scalable data pipelines on AWS using PySpark and Python. You will build ETL/ELT workflows with Glue and Step Functions, and develop serverless solutions with Lambda.

You will also manage IaC with Terraform, and implement CI/CD with GitLab/GitHub Actions/Jenkins while ensuring security, monitoring, and data governance in a lakehouse environment.

Qualifications

• Strong hands-on PySpark and Python for data engineering.
• Experience developing ETL pipelines using AWS Glue.
• Proficiency with AWS Step Functions for workflow orchestration.
• Experience building serverless solutions using AWS Lambda.
• Hands-on Terraform for Infrastructure as Code (IaC).
• Experience implementing CI/CD pipelines with GitLab, GitHub Actions, Jenkins.
• Good understanding of AWS services, data lakes, IAM, S3, CloudWatch, and monitoring.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark on AWS.
• Build and orchestrate ETL/ELT workflows using AWS Glue and AWS Step Functions.
• Develop serverless applications and automation using AWS Lambda.
• Write clean, efficient PySpark/Python code following engineering best practices.
• Provision and manage cloud infrastructure using Terraform (IaC).
• Implement CI/CD pipelines to automate code deployment, testing, and infra changes.
• Monitor, troubleshoot, and optimize data pipelines for performance, cost, and reliability.
• Collaborate with business stakeholders to deliver data solutions.
• Follow DevOps, security, and coding standards throughout the engagement.

Skills

PySpark

Python

SQL

Data modelling

DBT

Airflow

Tools

Terraform

AWS Glue

AWS Step Functions

AWS Lambda

GitLab

GitHub Actions

Jenkins

IAM

S3

CloudWatch

Job description

This role requires strong data Engineering& analysis expertise across SQL, Data modelling, DBT, Airflow, and Lakehouse/data lake architecture, with hands-on experience querying large-scale distributed datasets directly from data lakes.

Banking/financial services experience, exposure to ledger/ transactional/ regulatory data, support for migration/modernization program, familiarity, with data governance / metadata. regulatory controls.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark on AWS.
• Build and orchestrate ETL/ELT workflows using AWS Glue and AWS Step Functions.
• Develop serverless applications and automation using AWS Lambda.
• Write clean, efficient, and maintainable Python/PySpark code following engineering best practices.
• Provision and manage cloud infrastructure using Terraform (Infrastructure as Code).
• Implement and maintain CI/CD pipelines to automate code deployment, testing, and infrastructure changes.
• Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and cost efficiency.
• Collaborate with business stakeholders to deliver data solutions.
• Follow DevOps, security, and coding standards throughout the engagement.

Required Skills

• Strong hands-on experience with PySpark and Python for data engineering.
• Experience developing ETL pipelines using AWS Glue.
• Proficiency with AWS Step Functions for workflow orchestration.
• Experience building serverless solutions using AWS Lambda.
• Hands-on experience with Terraform for Infrastructure as Code (IaC).
• Experience implementing CI/CD pipelines using tools such as GitLab, GitHub Actions, Jenkins, or similar.
• Good understanding of AWS services, data lakes, IAM, S3, CloudWatch, and monitoring.