Databricks Data Engineer

Avanade Inc.; Kuala Lumpur; On-site; MYR 150,000 - 200,000; Full time

Job summary

A global technology consulting firm in Kuala Lumpur seeks analytical data professionals to join their Data & AI Practice. Responsibilities include designing and maintaining ETL/ELT pipelines using Databricks, optimizing performance, and ensuring data quality. Ideal candidates will work with cross-functional teams to support analytics and machine learning, implementing CI/CD for automation and ensuring documentation for processes. Join a team that values data-driven decision-making and problem-solving.

Qualifications

• Proficiency in designing and maintaining ETL/ELT pipelines.
• Experience with Databricks and Spark job optimization.
• Familiar with data quality checks and governance policies.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Utilize Databricks features for data orchestration.
• Implement CI/CD and DataOps for automated deployment.

Skills

Data Pipeline Development

ETL/ELT pipelines

Databricks

PySpark

SQL

Data governance

Tools

Databricks Workflows

Delta Lake

Job description

SummaryOur talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly do what matters.Key responsibilitiesData Pipeline Development: Design, build, and maintain robust and scalable ETL/ELT pipelines using Databricks, PySpark/Scala, and SQL to ingest, transform, and load data from diverse sources (e.g., databases, APIs, streaming services) into Delta Lake.Databricks Ecosystem Utilization: Utilize core Databricks features such as Delta Lake, Databricks Workflows (or Jobs), Databricks SQL, and Unity Catalog for pipeline orchestration, data management, and governance.Performance Optimization: Tune and optimize Spark jobs and Databricks clusters for maximum efficiency, performance, and cost-effectiveness.Data Quality and Governance: Implement data quality checks, validation rules, and observability frameworks. Adhere to data governance policies and leverage Unity Catalog for fine-grained access control.Collaboration: Work closely with Data Scientists, Data Analysts, and business stakeholders to translate data requirements into technical solutions and ensure data is structured to support analytics and machine learning use cases.Automation & DevOps: Implement CI/CD and DataOps principles for automated deployment, testing, and monitoring of data solutions.Documentation: Create and maintain technical documentation for data pipelines, data models, and processes.Troubleshooting: Monitor production pipelines, troubleshoot complex issues, and perform root cause analysis to ensure system reliability and stability.

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Job candidates will not be obligated to disclose sealed or expunged records of conviction or arrest as part of the hiring process.

Avanade is committed to providing veteran employment opportunities to our service men and women.

Please read Avanade's Hiring and Privacy Statement for more information on how we process your data during the Recruiting and Hiring process.