Data Engineer (ADF & Databricks)

half the sky; Kuala Lumpur; On-site; MYR 150,000 - 200,000; Full time

Job summary

A leading technology service provider based in Kuala Lumpur is seeking a Data Engineer to support the implementation of data projects across various platforms. The role involves designing modern data pipelines, leveraging Azure Data Factory and Databricks, and ensuring data accessibility for analysis. Candidates should have around 5 years of experience in Data Engineering and be proficient in data governance and cloud transformation. The organization promotes a people-first culture with flexible working arrangements and competitive pay.

Benefits

Employee share purchasing

Flexible work arrangements

Diversity and inclusion commitment

Competitive pay

Qualifications

• Around 5 years of working experience as a Data Engineer.
• Demonstrated experience turning business requirements into technical solutions.
• Experience in business process mapping of data and analytics solutions.
• Ability to conduct data profiling and mapping for technical design.

Responsibilities

• Translate business requirements to technical solutions.
• Analyze and identify future business opportunities.
• Deliver architectures for modernizing enterprise data solutions.
• Design and build modern data pipelines.
• Develop and maintain data warehouse schematics.

Skills

Data Engineering

Azure Data Factory

Databricks

Data Governance

DevOps

Data Analysis

Python

Power BI

Tools

ETL/ELT Tools

Azure Cloud Technologies

PowerShell

Job description

Kuala Lumpur, Federal Territory of Kuala Lumpur, Malaysia

About the job 69028 Data Engineer (ADF & Databricks)

About Avanade

Avanade is the leading provider of innovative digital, cloud and advisory services, industry solutions and design-led experiences across the Microsoft ecosystem. Every day, our 59,000 professionals in 26 countries make a genuine human impact for our clients, their employees and their customers.

We have been recognized as Microsofts Global SI Partner of the Year more than any other company. With the most Microsoft certifications (60,000+) and 18 (out of 18) Gold‑level Microsoft competencies, we are uniquely positioned to help businesses grow and solve their toughest challenges.

We are a people first company, committed to providing an inclusive workplace where employees feel comfortable being their authentic selves. As a responsible business, we are building a sustainable world and helping people from underrepresented communities fulfil their potential.

Majority owned by Accenture, Avanade was founded in 2000 by Accenture LLP and Microsoft Corporation. Learn more at www.avanade.com .

As a Data Engineer, you will support the implementation of projects focused on collecting, aggregating, storing, reconciling, and making data accessible from disparate sources to enable analysis and decision making. This role will also play a critical part in the data supply chain, by ensuring stakeholders can access and manipulate data for routine and ad hoc analysis. Additionally, you will support the full lifecycle of data from ingesting through analytics to action.

Key Role Responsibilities:

Day‑to‑day you will:

• Translate business requirements to technical solutions leveraging strong business acumen
• Analyze current business practices, processes and procedures as well as identifying future business opportunities for leveraging Microsoft Azure Data & Analytics PaaS Services
• Support the planning and implementation of data design services, providing sizing and configuration assistance and performing needs assessments.
• Deliver architectures for transformations and modernizations of enterprise data solutions using Azure cloud data technologies.
• Design and Build Modern Data Pipelines and Data Streams.
• Design and Build Data Service APIs.
• Develop and maintain data warehouse schematics, layouts, architectures and relational/non‑relational databases for data access and Advanced Analytics.
• Expose data to end users using PowerBI, Azure API Apps or any other modern visualization platform or experience
• Implement effective metrics and monitoring processes.

Skill & Capability Requirements:

• Around 5 years of working experience as a Data Engineer
• Demonstrated experience in turning business use cases and requirements into technical solutions
• Experience in business processing mapping of data and analytics solutions.
• Ability to conduct data profiling, cataloging and mapping for technical design and construction of technical data flows
• The ability to apply such methods to solve business problems using one or more Azure Data and Analytics services in combination with building data pipelines, data streams, and system integration.
• Ability to leverage a variety of programming languages & data crawling/processing tools to ensure data reliability, quality & efficiency
• Experienced in Cloud Data Transformation using ETL/ELT tools such as Azure Data Factory, Databricks
• Experienced in Dev‑Ops processes (including CI/CD) and Infrastructure as code fundamentals.
• Experienced in Data Governance tools like Unity Catalog / Purview, Master Data Management (MDM) and Data Quality tools and processes
• Knowledge of business intelligence tools such as Power BI, Tableau, Qlik, Cognos TM1
• Knowledge in data modeling and database management, such as performance tuning of the Enterprise Data Warehouse, Data Mart, and Business Intelligence Reporting environments, and support the integration of those systems with other applications
• Knowledge of Python is a plus.
• Experience preparing data for Data Science and Machine Learning.
• Experience preparing data for use in Azure Machine Learning / Azure Databricks is a plus.
• Experience preparing data and building data pipelines for AI Use Cases (text, voice, image, etc).
• Designing and building Data Pipelines using streams of IoT data.
• Knowledge of Lambda and Kappa architecture patterns.
• Strong team collaboration and experience working with remote teams.
• Working experience in Visual Studio, PowerShell Scripting, and ARM templates.
• Azure certifications (AZ‑900, AZ‑200, and 201), Databricks certification, DevOps certifications

What we offer you

Come for the distinctive experiences you have helping forward‑thinking corporations, non‑profits, and governments push the boundaries of digital innovation. Stay for the limitless learning opportunities that encourage you to master Microsoft and pursue big ideas. Enjoy ambitious growth for yourself as part of Avanade's people‑first culture with benefits like employee share purchasing, flexible work arrangements, a commitment to diversity and inclusion, and competitive pay.