Databricks ML Lead: Lead Design & Hands-on ML Pipelines

Avanade Inc.; Kuala Lumpur; On-site; MYR 100,000 - 140,000; Full time

Job summary

A leading technology firm in Kuala Lumpur seeks a Databricks M/L Technical Lead to join their data practice. In this senior role, you will design and develop scalable data solutions on the Databricks Lakehouse Platform. Responsibilities include defining data architectures, optimizing performance for Spark jobs, and leading a team of engineers in maintaining high coding standards. This position requires expertise in PySpark, Scala, SQL, and a strong understanding of the Databricks ecosystem.

Qualifications

• Experience in designing data architectures using the Databricks ecosystem.
• Ability to write high-quality, production-grade code in PySpark/Scala and SQL.
• Experience with performance tuning for large-scale Spark jobs.

Responsibilities

• Define and implement robust data architectures.
• Write complex batch and streaming ETL/ELT pipelines.
• Lead performance tuning and optimization efforts.
• Define and enforce code quality and technical standards.

Skills

PySpark

Scala

SQL

Data architecture design

Performance optimization

Tools

Databricks

Delta Lake

Unity Catalog

Job description

A leading technology firm in Kuala Lumpur seeks a Databricks M/L Technical Lead to join their data practice. In this senior role, you will design and develop scalable data solutions on the Databricks Lakehouse Platform. Responsibilities include defining data architectures, optimizing performance for Spark jobs, and leading a team of engineers in maintaining high coding standards. This position requires expertise in PySpark, Scala, SQL, and a strong understanding of the Databricks ecosystem.