Databricks Data Engineer

Avanade Inc.; Kuala Lumpur; On-site; MYR 70,000 - 90,000; Full time

Job summary

A leading technology consulting firm in Kuala Lumpur is seeking an experienced data professional to join their Data & AI Practice. You will design scalable ETL/ELT pipelines, optimize Spark jobs, and work closely with data scientists to drive data-driven decisions. This role offers the opportunity to expand your skills in a collaborative environment focused on innovation and excellence.

Responsibilities

• Design and maintain ETL/ELT pipelines using Databricks, PySpark/Scala, and SQL.
• Utilize Databricks features for data management and governance.
• Optimize Spark jobs and Databricks clusters for efficiency.
• Implement data quality checks and adhere to governance policies.
• Translate data requirements into technical solutions with stakeholders.
• Implement CI/CD and DataOps for automated testing and monitoring.
• Create technical documentation for data processes.
• Monitor and troubleshoot production pipelines.

Skills

ETL/ELT pipeline development

Databricks

PySpark/Scala

SQL

Data governance

Performance optimization

CI/CD

DevOps

Job description

Summary

Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly do what matters.

Key Responsibilities

• Data Pipeline Development: Design, build, and maintain robust and scalable ETL/ELT pipelines using Databricks, PySpark/Scala, and SQL to ingest, transform, and load data from diverse sources (e.g., databases, APIs, streaming services) into Delta Lake.
• Databricks Ecosystem Utilization: Utilize core Databricks features such as Delta Lake, Databricks Workflows (or Jobs), Databricks SQL, and Unity Catalog for pipeline orchestration, data management, and governance.
• Performance Optimization: Tune and optimize Spark jobs and Databricks clusters for maximum efficiency, performance, and cost-effectiveness.
• Data Quality and Governance: Implement data quality checks, validation rules, and observability frameworks. Adhere to data governance policies and leverage Unity Catalog for fine‑grained access control.
• Collaboration: Work closely with Data Scientists, Data Analysts, and business stakeholders to translate data requirements into technical solutions and ensure data is structured to support analytics and machine learning use cases.
• Automation & DevOps: Implement CI/CD and DataOps principles for automated deployment, testing, and monitoring of data solutions.
• Documentation: Create and maintain technical documentation for data pipelines, data models, and processes.
• Troubleshooting: Monitor production pipelines, troubleshoot complex issues, and perform root cause analysis to ensure system reliability and stability.

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Additional Information

Job candidates will not be obligated to disclose sealed or expunged records of conviction or arrest as part of the hiring process.

Avanade is committed to providing veteran employment opportunities to our service men and women.

Please read Avanade's Hiring and Privacy Statement for more information on how we process your data during the Recruiting and Hiring process.