Senior Data Engineer - Capital Markets

Macquarie Group; Indonesia; On-site; IDR 400,000,000 - 900,000,000; Full time

Job summary

Macquarie Group is seeking an experienced data/software engineer in Indonesia with 8+ years of hands-on experience and strong Python skills. You will build scalable ETL pipelines, implement data models, and optimize Spark/PySpark workloads in a cloud environment using AWS, Redshift, S3 and Glue.

The role emphasizes governance, security, and collaboration in an agile setting, with production support and observability responsibilities.

Qualifications

• 8+ years in data or software engineering with Python focus.
• Strong SQL using joins, CTEs, window functions.
• Experience building scalable ETL pipelines and data models.
• Spark or PySpark experience with performance tuning.
• AWS experience (Redshift, S3, Glue, IAM, Kinesis); familiarity with GCP.
• Airflow, Argo or dbt with scheduling, retries and backfills.
• Metadata, lineage, reconciliation, data protection and auditability.
• Docker, Unix/Linux; Kubernetes is a plus.
• Git, CI/CD, testing, observability and production support.
• Awareness of AI-assisted tools like GitHub Copilot; data privacy.

Responsibilities

• Develop and maintain scalable data pipelines and transformations.
• Collaborate with stakeholders to resolve data issues and risks.
• Implement governance, lineage and auditability across pipelines.

Skills

Python

SQL

ETL pipelines

Spark

PySpark

AWS

Redshift

S3

Glue

IAM

Kinesis

GCP

SQS

SNS

Kafka

Airflow

dbt

Kubernetes

Docker

Unix/Linux

Git

CI/CD

Observability

AI tools

Copilot

Embeddings

Vector search

Bedrock

Agile

Stakeholder engagement

Financial services

Tools

Spark

PySpark

Docker

Unix/Linux

Git

dbt

Job description

• 8+ years’ experience in data or software engineering, with strong Python skills and a record of delivering maintainable enterprise solutions.
• Advanced SQL skills, including joins, common table expressions and window functions.
• Experience building scalable extract, transform and load pipelines, reusable transformations and relational or dimensional models, including incremental loading, schema evolution and batch or streaming integration.
• Experience with Spark or PySpark, including performance optimisation and cloud cost awareness.
• Hands‑on experience with Amazon Web Services, including Redshift, S3, Glue, Identity and Access Management and Kinesis, plus familiarity with Google Cloud and SQS, SNS or Kafka.
• Experience with Airflow, Argo or dbt, including dependency management, scheduling, retries and backfills.
• Experience implementing metadata, lineage, reconciliation and completeness checks, with secure access, data protection and auditability.
• Working knowledge of Docker and Unix or Linux, with Kubernetes experience advantageous.
• Experience with Git, code review, automated testing, continuous integration and deployment, controlled releases, observability, recoverable processing and production support.
• Familiarity with approved artificial intelligence‑assisted engineering tools such as GitHub Copilot, with independent validation and protection of confidential data; exposure to document processing, embeddings, vector search, retrieval‑augmented generation and Amazon Bedrock or equivalent is advantageous.
• Strong stakeholder engagement skills, including investigating data issues, challenging assumptions and communicating dependencies and risks; experience in agile teams, knowledge sharing and financial services or trading is advantageous.
• 8+ years’ experience in data or software engineering, with strong Python skills and a record of delivering maintainable enterprise solutions.
• Advanced SQL skills, including joins, common table expressions and window functions.
• Experience building scalable extract, transform and load pipelines, reusable transformations and relational or dimensional models, including incremental loading, schema evolution and batch or streaming integration.
• Experience with Spark or PySpark, including performance optimisation and cloud cost awareness.
• Hands‑on experience with Amazon Web Services, including Redshift, S3, Glue, Identity and Access Management and Kinesis, plus familiarity with Google Cloud and SQS, SNS or Kafka.
• Experience with Airflow, Argo or dbt, including dependency management, scheduling, retries and backfills.
• Experience implementing metadata, lineage, reconciliation and completeness checks, with secure access, data protection and auditability.
• Working knowledge of Docker and Unix or Linux, with Kubernetes experience advantageous.
• Experience with Git, code review, automated testing, continuous integration and deployment, controlled releases, observability, recoverable processing and production support.
• Familiarity with approved artificial intelligence‑assisted engineering tools such as GitHub Copilot, with independent validation and protection of confidential data; exposure to document processing, embeddings, vector search, retrieval‑augmented generation and Amazon Bedrock or equivalent is advantageous.
• Strong stakeholder engagement skills, including investigating data issues, challenging assumptions and communicating dependencies and risks; experience in agile teams, knowledge sharing and financial services or trading is advantageous.