Senior Data Engineer

Kuala Lumpur Kepong Berhad

Petaling Jaya

In loco

MYR 150.000 - 230.000

Tempo pieno

2 giorni fa

Candidati tra i primi

Una candidatura completa in un minuto

Curriculum e lettera di presentazione personalizzati, pronti da inviare.

Supera i filtri ATS

Descrizione del lavoro

Kuala Lumpur Kepong Berhad Group is seeking a seasoned Data Management & Technical Development professional to oversee end-to-end database ecosystems (MS SQL Server, PostgreSQL, DuckDB) and drive scalable analytics solutions. You will design data pipelines, optimize performance, and collaborate with business stakeholders to align data platforms with strategic goals.

Responsibilities include building and maintaining data models (star and snowflake), implementing ingestion strategies, monitoring

Competenze

Minimum of 5 years in business solutioning, data/analytics consulting or data warehouse delivery.
Understanding of dimensional modelling (star & snowflake) and SCD concepts.
Proficiency in MS SQL Server including T-SQL, SSIS packages.
Experience with SAP ECC6 or SAP S/4HANA is a plus.

Mansioni

Lead end-to-end management of MS SQL Server, PostgreSQL and DuckDB databases for stability, scalability and reliability.
Design and implement high-quality, low-latency data solutions and scripts to extract data from SAP ECC6.
Implement batch and stream data ingestion strategies considering source constraints.
Write and optimize complex T-SQL queries, stored procedures, views, functions and triggers.
Orchestrate and schedule complex data workflows for timely data delivery.
Monitor daily job statuses and troubleshoot recovery as needed.
Develop and maintain staging structures and star/snowflake data models in SQL Server.
Contribute to data quality validation, reconciliation checks and exception handling.
Document pipeline logic, data lineage, transformation rules and data dictionaries.

Conoscenze

Data warehousing concepts

SQL development

Stakeholder collaboration

Performance tuning

Data governance

Formazione

Bachelor's degree in IT/CS/Data Analytics/Engineering

Strumenti

SSIS

SSAS

SQL Server

PowerShell

DuckLake

Delta Lake

Descrizione del lavoro

Role and Responsibilities

A.DATA MANAGEMENT & TECHNICAL DEVELOPMENT

Lead the end-to-end management of MS SQL Server, PostgreSQL and DuckDB (DuckLake) database systems, ensuring their stability, scalability and reliability.
Apply best practices to design and implement high quality, low latency and scalable data solutions and scripts to extract data from source system (SAP ECC6).
Implement data ingestion strategies (batch and stream) by factoring in source-side constraints and data availability.
Write, optimize and troubleshoot complex T-SQL queries, stored procedures, views, functions and triggers to support data transformation, loading, and validation logic within MS SQL Server, in line with established coding and naming standards.
Orchestrate and schedule complex data workflows, ensuring timely and reliable data delivery.
Implement and manage monitoring solutions to proactively address and prevent performance issues. Monitor daily job statuses and perform troubleshoot and recovery when necessary.
Develop and maintain staging structures and data models (star and snowflake schemas) within MS SQL Server, ensuring data is structured for efficient querying and reliable analytical consumption.
Contribute to data quality validation, reconciliation checks, and exception handling across data pipelines, ensuring completeness, accuracy, and consistency of data from source to destination.
Document pipeline logic, data lineage, transformation rules, table definitions and data dictionary entries, ensuring technical artefacts are kept current and accessible to the team.

B.OPTIMIZATION & PLATFORM TRANSFORMATION

Deep understanding and assessment of SSIS data pipelines to refactor SQL scripts and migrate them to the new data lakehouse environment.
Lead in schema design, table optimization (clustering, partitioning, indexing) and other optimization strategies to handle growth.
Conduct in-depth performance tuning activities, optimizing SQL queries, and database configurations for overall efficiency.
Identify opportunities to optimize data processes, reduce complexities and costs.
Participate in technical discussions, debates, peer reviews and knowledge-sharing sessions on topics related to the data warehouse.
Contribute to the establishment of a data governance framework. Drive the standards and strategy for maintaining a proper and traceable governance model for all end-to-end data assets.

C.BUSINESS PARTNERSHIP & PROJECT MANAGEMENT

Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
Assess and evaluate Business Requirement Study (BRS) to ensure that they are thoroughly scoped, covering objectives, business value, data sourcing implications and delivery timeline.
Serve as the primary liaison for users when it comes to data related issues. Troubleshoot and debug issues arising from user endpoints.

Job Success Requirements

Minimum of 5 years' experience in business solutioning, data / analytics consulting or a data warehouse delivery role, with a track record of partnering directly with business stakeholders.
Understanding of data warehousing concepts, including dimensional modelling (star & snowflake schemas), slowly changing dimensions (SCD), and staging/ODS design patterns, with the ability to apply these principles to schema design and pipeline architecture decisions.
Proficiency in MS SQL Server, including database design, T-SQL development (stored procedures, views, functions, triggers), query optimisation, performance tuning, developing and maintaining SSIS packages for data pipelines.
Good to have: SAP ECC6 or SAP S/4HANA, table formats (e.g. Delta Lake, Apache Iceberg, Apache Hudi, Ducklake), file formats (e.g. Parquet, Avro), data platforms (e.g. Snowflake, Databricks, BigQuery, Synapse), columnar serialization interchange formats (e.g. Apache Arrows)

Technology Stack We Use