Data Engineer: AI-Driven Pipelines on Databricks

Theaccessgroup

Kuala Lumpur

Hybrid

MYR 80,000 - 120,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Wellbeing days - 25 days leave
Health contribution
Charity days

Job summary

The Access Group is seeking a passionate Data Engineer to design, build, and operate scalable data pipelines on Databricks, ingesting data from internal and external APIs. You will develop dimensional data models to support analytics and downstream applications, while applying modern CI/CD practices and AI-assisted development tools.

You will collaborate with product, engineering, analytics, and business teams to deliver robust end-to-end data solutions, leveraging technologies like Python, SQL,

Qualifications

  • 3-5 years of professional experience in Data Engineering.
  • Strong hands-on experience with Databricks, including Spark/PySpark, SQL, Delta Lake, and Databricks Workflows.
  • Experience building and supporting API-driven data ingestion pipelines.
  • Solid understanding of dimensional modelling and layered/medallion data architectures.
  • Experience with data engineering frameworks and tools, including orchestration, testing, CI/CD, and Git-based development practices.
  • Strong programming skills in Python and SQL.
  • Experience working with structured and unstructured data sources.
  • Understanding of AI-assisted development and practical use of AI in data engineering workflows.
  • Strong communication and stakeholder management skills.
  • Desirable Experience with AI-assisted development tools such as Claude Code, OpenAI Codex, GitHub Copilot CLI, or Devin.
  • Experience with NoSQL technologies and non-relational data stores.
  • Exposure to Azure services, including Entra ID and Azure DevOps.
  • Experience handling observability, monitoring, security, vulnerability, or infrastructure-related datasets.
  • Familiarity with Business Intelligence platforms and reporting tools.
  • Experience developing lightweight services or APIs using frameworks such as FastAPI.

Responsibilities

  • Design, build, and operate reliable data pipelines on Databricks from ingestion to curated layers.
  • Ingest data from internal and third-party APIs with authentication, pagination, and schema evolution.
  • Develop and maintain scalable data models using dimensional/medallion architectures.
  • Apply modern data engineering practices: modular development, automated testing, orchestration, CI/CD, version control.
  • Monitor pipeline performance, quality, reliability, and cost efficiency; resolve issues proactively.
  • Collaborate with product, engineering, analytics and business teams for end-to-end solutions.
  • Leverage AI-assisted development tools to accelerate coding, testing, and documentation while ensuring quality.
  • Identify opportunities for automation and continuous improvement across the data platform.
  • Contribute to engineering standards,Documentation, peer reviews and knowledge sharing.

Skills

Databricks
Spark/PySpark
Python
SQL
Delta Lake
APIs
CI/CD
Git
AI-assisted development
Data modelling
Neo4j
NoSQL
Azure DevOps
FastAPI
Graph databases

Tools

Azure DevOps
Neo4j
Git-based development

Job description

The Access Group is seeking a passionate Data Engineer to design, build, and operate scalable data pipelines on Databricks, ingesting data from internal and external APIs. You will develop dimensional data models to support analytics and downstream applications, while applying modern CI/CD practices and AI-assisted development tools.

You will collaborate with product, engineering, analytics, and business teams to deliver robust end-to-end data solutions, leveraging technologies like Python, SQL,

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer — AI-Driven Data Pipelines (Databricks)
Data Engineer — AI-Driven Data Pipelines (Databricks)

Access UK • Kuala Lumpur

Hybrid
MYR 120,000 - 180,000
Health contribution
25 days leave
Charity days
+1
Databricks Data Engineer — Cloud Pipelines & Data Products
Databricks Data Engineer — Cloud Pipelines & Data Products

EPAM Systems Inc • Malaysia

On-site
MYR 80,000 - 140,000
Hybrid Data Engineer – Kuala Lumpur, AI‑Driven Pipelines
Hybrid Data Engineer – Kuala Lumpur, AI‑Driven Pipelines

The Access Group • Kuala Lumpur

Hybrid
MYR 120,000 - 210,000
25 days leave
Health contribution
3 charity days allocated
Databricks Data Integration Lead
Databricks Data Integration Lead

EPAM Systems Inc • Kuala Lumpur

On-site
MYR 180,000 - 300,000
Data Engineer: AI-Powered Pipelines & Scalable DataOps
Data Engineer: AI-Powered Pipelines & Scalable DataOps

Thakral One Pte. Ltd. • Kuala Lumpur

On-site
MYR 60,000 - 120,000
Data Engineer (Azure Databricks)
Data Engineer (Azure Databricks)

EPAM Systems • Malaysia

On-site
MYR 80,000 - 130,000
Data Engineer: AI-Driven Pipelines & Cloud Data Mastery
Data Engineer: AI-Driven Pipelines & Cloud Data Mastery

Thakral One Pte. Ltd. • Kuala Lumpur

On-site
MYR 120,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Agensi Pekerjaan Penta Consultancy Sdn. Bhd • Kuala Lumpur

On-site
MYR 180,000 - 240,000
Azure Databricks Data Engineer for Analytics & Power BI
Azure Databricks Data Engineer for Analytics & Power BI

EPAM Systems • Kuala Lumpur

On-site
MYR 90,000 - 180,000
Senior Data Engineer: Databricks-powered ETL & Pipelines
Senior Data Engineer: Databricks-powered ETL & Pipelines

Averis Sdn Bhd • Kuala Lumpur

On-site
MYR 120,000 - 240,000
Medical benefits
Training and development
Overseas postings
+5