Azure Data Engineer: Cloud Pipelines, Spark & APIs

Inter Island Group; Selangor; On-site; MYR 120,000 - 180,000; Full time

Job summary

Inter Island Group in Malaysia is seeking an experienced Data Engineer to join the Automotive Industry, building modern cloud-based data solutions on Microsoft Azure and Azure Databricks.

You will develop and maintain data pipelines with Azure Databricks, Python, Spark, and manage Databricks workflows, ADLS, Hive Metastore, and compute clusters, while implementing CI/CD with Azure DevOps and GitHub Actions.

Qualifications

• 7+ years of data engineering experience.
• Hands-on experience with Azure Databricks, Python, and Azure DevOps.
• Strong knowledge of Spark and SQL.
• Experience with Microsoft Azure components (ADLS, Key Vault, Entra ID).
• CI/CD, GitHub/Azure Repos, and YAML pipelines.
• Experience with FastAPI, REST APIs, and Docker.
• Cloud-native data engineering mindset.

Responsibilities

• Develop and maintain data pipelines using Azure Databricks, Python, and Spark.
• Manage Databricks workflows, ADLS, Hive Metastore, and compute clusters.
• Develop and maintain CI/CD pipelines using Azure DevOps and GitHub Actions.
• Work with Azure Key Vault, Entra ID, and Azure Log Analytics.
• Develop and integrate REST APIs using FastAPI.
• Build and deploy containerized applications using Docker and Azure Container Registry.
• Work with NATS and Redis for messaging and caching.
• Collaborate with project teams using SCRUM.

Skills

Data engineering

Scrum

Tools

Azure Databricks

Python

Spark

Azure DevOps

GitHub Actions

Azure Key Vault

Docker

FastAPI

REST APIs

NATS

Redis

Azure Container Registry

SQL

Entra ID

ADLS

Hive Metastore

Job description

Inter Island Group in Malaysia is seeking an experienced Data Engineer to join the Automotive Industry, building modern cloud-based data solutions on Microsoft Azure and Azure Databricks.

You will develop and maintain data pipelines with Azure Databricks, Python, Spark, and manage Databricks workflows, ADLS, Hive Metastore, and compute clusters, while implementing CI/CD with Azure DevOps and GitHub Actions.