senior data engineer

RecruitFirst Pte Ltd; Malaysia; On-site; MYR 89,000 - 112,000; Full time

Job summary

RecruitFirst Pte Ltd in Malaysia is seeking a Senior Data Engineer to drive regional BI & Analytics initiatives for a multinational F&B client. You will design, build, and optimize data pipelines across CRM, sales and D365 sources using Azure Databricks and Synapse Analytics.

You will implement star schemas, ensure data quality and security, and collaborate with data analysts and business stakeholders to translate requirements into scalable data solutions across Southeast Asia.

Qualifications

• Bachelor’s degree in Computer Science, Data Science, Statistics, Mathematics or equivalent.
• ,
• ],
• job_responsibilities_description":[

Responsibilities

• Design, develop, and maintain robust data pipelines and ETL processes to support data integration and transformation across multiple data sources.

Skills

Python

Scala

R

T-SQL

MDX

DAX

Data analysis

Data mining

ETL

Education

Bachelor's degree in CS/DS/Statistics/Math

Tools

Delta Lake

Databricks

Azure Synapse Analytics

Job description

Overview

Job-type Full-Time Job Category IT - Software Industries IT Salary MYR 8,000 - 10,000 /Month

Who you'll be working for

Client is a multinational company in F&B.

What requirements you'll need to be eligible

• Bachelor’s degree holder in Computer Science, Data Science, Statistics, Mathematics or equivalent
• Minimum 5 years of experience in data engineering or a similar role.
• Strong proven expertise in Azure Synapse Analytics, Databricks, and other Azure cloud technologies.
• Proficiency in Python, PySpark, and other relevant programming languages for ETL automation.
• Experience in designing and implementing data models, including star schemas and other dimensional models.
• Experience in Data Management tools such as Data Lake, BI, ETL, Dashboards, Balanced Scorecards, etc.
• Excellent communication and presentation skills, to communicate complex analytical and technical content clearly and effectively.
• Demonstrated ability to work with large, complex datasets and build scalable data solutions.
• Track record of forming strong partnerships with cross-functional teams.
• Experience with data visualization tools and an understanding of how data engineering supports data analytics.

KNOWLEDGE, SKILLS & ABILITIES REQUIRED

• Highly proficient in data analysis and data mining skills (such as Python, Scala, R, T-SQL, MDX, DAX, etc.)
• Strong analytical skills with the ability to translate business needs into technical solutions.
• In-depth knowledge and understanding of data relationships and able to comprehend the data.
• Understand the reporting and business intelligence best practices.
• Ability to synthesize information from various sources to Data Management Platform to create a single and actionable point of view and direction for BI reporting requirements.
• Highly resourceful, with strong research skills and a working knowledge of digital marketing analytics.
• Possess a strong ability to query and use large, robust data sets across the latest Delta Lake and Databrick platforms.
• Strong understanding of data governance, security, and compliance best practices.
• Ability to manage multiple priorities and projects simultaneously.

OTHERS

• Fluent communication in English and interpersonal skills, both verbal and written.
• Meticulous and good team player who can work with various levels of stakeholders from different cultures and locations.
• Ability to work in a flexible, dynamic, and fast-paced environment managing multiple projects concurrently.
• Should be able to travel to other countries if needed.

What you'll be doing on the job

An exciting opportunity to join a market leader in the F&B industry as Senior Data Engineer. You will get a full-fledged IT project implementation experience of regional BI & Analytics solutions across Southeast Asia Markets.

We are seeking a skilled Senior Data Engineer to join our team and contribute to the successful implementation of our regional Data & Analytics solutions across Southeast Asia Markets. This role will be pivotal in designing, building, and optimizing our data pipelines and infrastructure, ensuring seamless integration and processing of data from various sources, including CRM, sales, and D365 systems. The ideal candidate will have substantial experience in working with Azure technologies, particularly Databricks and Synapse Analytics, and will be proficient in automating ETL processes using SQL, Python or PySpark.

• Design, develop, and maintain robust data pipelines and ETL processes to support data integration and transformation across multiple data sources.
• Design analytical solutions for strategic business problems through engagement and research.
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and translate them into technical solutions in meeting milestones, timelines, and deliverables for projects.
• Implement and optimize data models, including star schemas, to facilitate efficient data querying and analysis.
• Leverage Azure Synapse Analytics, Databricks, and other Azure services to build scalable data infrastructure.
• Ensure data quality and consistency across all layers of the data lakehouse (bronze, silver, and gold layers).
• Automate ETL processes using SQL, Python, PySpark, or other relevant technologies to streamline data ingestion and transformation.
• Monitor and maintain the performance, security, and reliability of data systems.
• Provide technical leadership and guidance on best practices for data engineering within the organization.
• Document data pipelines, processes, and architecture to ensure transparency and knowledge sharing.
• Work either individually or in project teams as a subject matter expert to deliver business needs.
  New data integration and data infrastructure
• A key contributor to the development of organization data infrastructure and collaborate closely with vendor’s data engineers on a day-to-day basis to ensure development drives business outcomes.
• Create impeccable and user-friendly documentation for data sources and data dictionaries across the company.
• Establish a clear Business Glossary to keep everyone on the same page by defining common terminology and interpretation of the same data.
• Validate data structure in the Data Management Platform, refine logic and data transformation, and build aggregation tables to provide seamless reporting and data access across the business.
• Improve data maturity with high data quality, reliability, and visibility of business insights in a consistent manner and enable the business team to identify opportunities and threats for the company.