Senior Data Engineer

KLK OLEO; Selangor; On-site; MYR 120,000 - 190,000; Full time

Job summary

KLK OLEO in Malaysia is seeking a senior Data Engineer to manage MS SQL Server, PostgreSQL and DuckDB, ensuring stable, scalable data solutions for analytics.

You will design data ingestion from SAP ECC6, orchestrate complex data workflows, enforce governance, optimize performance, and collaborate with business stakeholders to deliver a trusted data platform. A strong background in data modeling and lakehouse technologies is required.

Qualifications

• Minimum of 5 years in data/analytics consulting or a data warehouse delivery role with direct stakeholder partnership.
• Experience with SAP ECC6 or SAP S/4HANA, including ABAP data extraction approaches and table structures.
• Proficiency in MS SQL Server including T-SQL development, stored procedures, views, functions and triggers.
• Experience with data lakehouse formats (Delta Lake, Apache Iceberg, Apache Hudi, DuckLake) and related catalogs.
• Knowledge of dimensional modelling (star/snowflake), SCD, staging/ODS patterns and pipeline architecture.
• Hands-on SSIS and data orchestration (ADF) for ETL; strong data quality and governance focus.
• Structured documentation, data lineage, and data dictionary maintenance.

Responsibilities

• Lead end-to-end data management and platform development across SQL Server, PostgreSQL, and DuckDB.
• Provide design and optimization for data ingestion from SAP ECC6 and other sources, ensuring low latency and reliability.
• Develop and maintain data models, governance policies, and robust ETL pipelines; monitor and troubleshoot data workflows.
• Collaborate with business stakeholders to translate requirements into scalable technical solutions and roadmaps.
• Drive data lakehouse modernization efforts and contribute to cross-functional knowledge sharing and best practices.

Skills

SQL performance tuning

Data modeling

Data governance

RBAC & data masking

Education

Bachelor's degree in Information Technology, Computer Science, Data Analytics, Engineering or related STEM field

Tools

MS SQL Server

PostgreSQL

DuckDB

DuckLake

Snowflake

Databricks

BigQuery

Synapse

SSIS

ADF

PowerShell

Python

Job description

THIS POSITION IS TO BE BASED IN MUTIARA DAMANSARA, PETALING JAYA

ROLE AND RESPONSIBILITIES

A. DATA MANAGEMENT & TECHNICAL DEVELOPMENT

Lead the end-to-end management of MS SQL Server, PostgreSQL and DuckDB (DuckLake) database systems, ensuring their stability, scalability and reliability.

Monitor daily job statuses and perform troubleshoot and recovery when necessary.

Apply best practices to design and implement high quality, low latency and scalable data solutions and scripts to extract data from source system (SAP ECC6), using appropriate connectors, aligning with business requirements.

Implement data ingestion strategies (batch and stream) by factoring in source-side constraints and data availability.

Write, optimize and troubleshoot complex T-SQL queries, stored procedures, views, functions and triggers to support data transformation, loading, and validation logic within MS SQL Server, in line with established coding and naming standards.

Orchestrate and schedule complex data workflows, ensuring timely and reliable data delivery.

Implement and manage monitoring solutions to proactively address and prevent performance issues.

Implement and enforce data governance and security policies, such as RBAC and data masking.

Develop and maintain staging structures and data models (star and snowflake schemas) within MS SQL Server, ensuring data is structured for efficient querying and reliable analytical consumption.

Contribute to data quality validation, reconciliation checks, and exception handling across data pipelines, ensuring completeness, accuracy, and consistency of data from source to consumption layer in alignment with the team's Single Source of Truth principles.

Document pipeline logic, data lineage, transformation rules, table definitions and data dictionary entries, ensuring technical artefacts are kept current and accessible to the team.

B. OPTIMIZATION & PLATFORM TRANSFORMATION

Deep understanding and assessment of SSIS data pipelines to refactor SQL scripts and migrate them to the new data lakehouse environment.

Lead discussions and debates on technical and architectural topics that are related to the data warehouse.

Participate in peer reviews and knowledge-sharing sessions, actively contributing to a culture of continuous improvement and cross-training across both legacy and modern data tooling within the team.

Identify opportunities to optimize data processes, reduce complexities and costs.

Conduct in-depth performance tuning activities, optimizing SQL queries, and database configurations for maximum efficiency.

Optimize storage usage, query performance and overall data platform efficiency.

Lead in schema design, table optimization (clustering, partitioning, indexing) and other optimization strategies to handle growth

Contribute to the establishment of a data governance framework. Drive the standards and strategy for maintaining a proper and traceable governance model for all end-to-end data assets.

C. BUSINESS PARTNERSHIP & PROJECT MANAGEMENT

Collaborate with stakeholders to understand business requirements and translate them into technical solutions.

Assess, evaluate and prioritize new and change requests to ensure they are evaluated for strategic fit and feasibility before assigning them to the development pipeline.

Assess and evaluate Business Requirement Study (BRS) to ensure that they are thoroughly scoped, covering objectives, business value, data sourcing implications and delivery timeline.

Build and maintain strong relationships with departmental stakeholders to ensure alignment of objectives and a shared understanding of requirements.

Provide clear and timely communication on project status, risks and progress, keeping business stakeholders and ITBS management well-informed throughout the delivery lifecycle.

Serve as the primary liaison for users when it comes to data related issues. Troubleshoot and debug issues arising from user endpoints.

Lead engagement with business and operational units - including roadshows, workshops and discovery sessions.

Champion the adoption of the data and self-service analytics tools.

Job Success Requirements

Minimum of 5 years' experience in business solutioning, data / analytics consulting or a data warehouse delivery role, with a track record of partnering directly with business stakeholders.

Working knowledge of SAP ERP systems (SAP ECC6 or SAP S/4HANA), including familiarity with key functional modules (e.g. FI, CO, MM, SD), ABAP data extraction approaches (BAPI, IDOC, Open SQL, ABAP reports), and the ability to navigate SAP table structures for data sourcing purposes.

Experience in developing and maintaining a data lakehouse Table Formats (e.g. Delta Lake, Apache Iceberg, Apache Hudi, Ducklake), File Formats (e.g. Parquet, Avro), Catalogues (e.g. Unity Catalog, Hive, Ducklake) and Platforms (e.g. Snowflake, Databricks, BigQuery, Synapse).

Experience in advanced system design and data optimization using in-memory columnar formats such as Apache Arrows. Understanding of vectorized execution and query plans dependencies (e.g. scanning volume, predicate pushdowns, column pruning and the execution engine).

Understanding of data warehousing concepts, including dimensional modelling (star & snowflake schemas), slowly changing dimensions (SCD), and staging/ODS design patterns, with the ability to apply these principles to schema design and pipeline architecture decisions.

Proficiency in MS SQL Server, including database design, T-SQL development (stored procedures, views, functions, triggers), query optimisation and performance tuning.

Hands-on experience developing and maintaining SSIS packages for ETL processes, covering data extraction from heterogeneous sources, transformation logic, error handling, logging, and loading into SQL Server targets — with the ability to debug and tune existing packages independently.

Detail-oriented and methodical, with a structured approach to documentation, data quality validation, and issue escalation, combined with the ability to manage multiple concurrent deliverables and work effectively in a fast-paced, cross-functional team environment.

Language: SQL (mandatory), PowerShell, Python (bonus)

Orchestration: SSIS, ADF

Transformation: SQL, SSIS

Storage: SQL Server, ADLS2

Qualification

Bachelor's degree in Information Technology, Computer Science, Data Analytics, Engineering or a related STEM field.

Additional Notes

Maintains awareness of current developments in data and AI technologies, and able to evaluate their potential application within the organisation.