Data Engineer — AI-Driven Data Pipelines (Databricks)

Access UK; Kuala Lumpur; Hybrid; MYR 120,000 - 180,000; Full time

Benefits offered by this job

Health contribution

25 days leave

Charity days

Hybrid office

Job summary

Access Group is seeking a passionate Data Engineer to design, build, and operate scalable data pipelines on Databricks, ingesting data from APIs and shaping models with dimensional and medallion architectures.

You will work with stakeholders across product, engineering, analytics, and business teams to deliver reliable, end-to-end data solutions, leveraging AI-assisted development while ensuring data quality and governance.

Qualifications

• 3-5 years of professional experience in Data Engineering.
• Strong hands-on experience with Databricks, including Spark/PySpark, SQL, Delta Lake, and Databricks Workflows.
• Experience building and supporting API-driven data ingestion pipelines.
• Solid understanding of dimensional modelling and layered/medallion data architectures.
• Experience with data engineering frameworks and tools, including orchestration, testing, CI/CD, and Git-based development practices.
• Strong programming skills in Python and SQL.
• Experience working with structured and unstructured data sources.
• Understanding of AI-assisted development and the practical use of AI in data engineering workflows, including development, testing, documentation, and validation.
• Strong communication and stakeholder management skills.

Responsibilities

• Design, build, and operate reliable data pipelines on Databricks, from ingestion through to curated, consumption-ready data layers.
• Ingest data from internal and third-party APIs, managing authentication, pagination, schema evolution, rate limiting, retries, and incremental loading.
• Develop and maintain scalable data models using dimensional and medallion architecture principles to support analytics, reporting, and downstream applications.
• Apply modern data engineering best practices, including modular development, automated testing, orchestration, CI/CD pipelines, version control, and environment promotion.
• Monitor data pipeline performance, quality, reliability, and cost efficiency, proactively resolving issues before they impact users.
• Collaborate with product, engineering, analytics, and business teams to deliver robust end-to-end data solutions.
• Leverage AI-assisted development tools and agentic workflows to accelerate coding, testing, documentation, and solution design while maintaining accountability for quality and outcomes.
• Identify opportunities for automation and continuous improvement across the data platform.
• Contribute to engineering standards, documentation, peer reviews, and knowledge-sharing initiatives.

Skills

Databricks

Spark/PySpark

SQL

Delta Lake

APIs ingestion

Dimensional modelling

Medallion architecture

Orchestration/CI/CD

Python

Data wrangling

Communication

Tools

Databricks

Azure DevOps

Git-based workflows

Neo4j

Job description

Access Group is seeking a passionate Data Engineer to design, build, and operate scalable data pipelines on Databricks, ingesting data from APIs and shaping models with dimensional and medallion architectures.

You will work with stakeholders across product, engineering, analytics, and business teams to deliver reliable, end-to-end data solutions, leveraging AI-assisted development while ensuring data quality and governance.