Senior Data Engineer - SQL/Hive/Impala (Banking/6-months)

D L Resources Pte Ltd; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

D L Resources Pte Ltd is seeking candidates to support a strategic data migration and reporting transformation initiative in Singapore. The project migrates legacy datamart sources to new data marts, requiring strong SAS, SQL and Python development skills and experience with Hadoop-based platforms.

The role involves data analysis, source-to-target mapping, development of scripts, validation of migrated outputs, and comprehensive data documentation, ensuring governance and quality across the data

Qualifications

• Strong SAS/SQL/Python scripting and data manipulation skills.
• Hands-on experience with Hue, Hive, Impala and Oozie.
• Experience in data warehousing, ETL development, data migration or data engineering projects.
• Experience working with large and complex datasets.
• Strong understanding of relational databases and data modelling concepts.

Responsibilities

• Data Analysis & Discovery: Analyse existing tables, reports, datasets and workflows.
• Source Mapping & Migration: Perform source-to-target mapping and document field mappings.
• Development & Automation: Develop SQL scripts and data extraction/transformation processes.
• Data Validation & Testing: Reconcile outputs and validate data accuracy, timeliness.
• Documentation & Governance: Produce mapping and data lineage docs and runbooks.

Skills

SAS

SQL

Python

Data analysis

Education

Bachelor's degree or higher in Computer Science, Information Systems, Engineering, Mathematics, Statistics or related

Tools

Hue

Hive

Impala

Oozie

Job description

We are seeking for candidate to support a strategic data migration and reporting transformation initiative. The objective of the project is to migrate existing tables, datasets, and reports currently sourced from a legacy datamart and manual data sources to newly established enterprise data marts

The successful candidates will be responsible for analysing existing data processing and reporting logic, mapping legacy and manual data sources to the new data marts, redeveloping scripts and workflows, validating migrated outputs, and producing comprehensive data documentation. This role requires strong SAS, SQL and Python development skills, data analysis capabilities, and experience working with Hadoop-based data platforms

Job responsibilities

Data Analysis & Discovery

• Analyse existing tables, reports, datasets, and data processing workflows currently sourced from legacy data marts and manual processes
• Review and understand existing scripts, transformation logic, data dependencies, and business rules
• Identify data sources, data lineage, reporting requirements, and current-state processes
• Engage business to understand existing reporting methodologies and requirements

Source Mapping & Migration

• Perform source-to-target mapping between legacy data marts/manual sources and new data marts
• Analyse and document field mappings, transformation rules, calculations, and business logic
• Identify gaps, inconsistencies, and opportunities for process optimisation during migration
• Develop migration specifications and ensure alignment with business requirements

Development & Automation

• Develop, enhance, and maintain SQL scripts using Hive and Impala engines via Hue
• Develop scripts for data extraction, transformation, validation, and reporting processes
• Rebuild existing tables, reporting datasets, and data preparation processes using the new data marts as source systems
• Configure and support workflow scheduling and automation using Oozie
• Ensure developed solutions are scalable, efficient, and maintainable

Data Validation & Testing

• Perform reconciliation and validation between outputs generated from current sources and outputs generated from the new data marts
• Validate data accuracy, completeness, consistency, and timeliness
• Investigate and resolve data discrepancies and defects identified during testing
• Support User Acceptance Testing (UAT) and business validation activities
• Ensure migrated reports and datasets meet defined business and technical requirements

Documentation & Governance

• Produce and maintain project artefacts, including:
• Source-to-target mapping documents
• Data lineage documentation
• Business and technical logic documentation
• Metadata documentation
• Data dictionaries
• Reconciliation and validation reports
• Technical specifications and operational runbooks
• Ensure documentation complies with enterprise data governance and documentation standards

Job requirement

• Strong SQL, SAS or Python scripting and data manipulation skills.
• Hands-on experience with: Hue, Hive, Impala, Oozie
• Experience in data warehousing, ETL development, data migration, or data engineering projects.
• Experience working with large and complex datasets.
• Strong understanding of relational databases and data modelling concepts.

Qualifications

• Degree in Computer Science, Information Systems, Engineering, Mathematics, Statistics, or a related discipline
• Minimum 3 years of relevant experience in data engineering, ETL development, data warehousing, reporting, or data migration projects

Data Management Skills

• Source-to-target mapping
• Data lineage analysis and documentation
• Metadata management
• Data dictionary creation and maintenance
• Data quality assessment and reconciliation
• Technical and functional documentation

Soft Skills

• Strong analytical and problem-solving abilities
• Ability to understand and reverse-engineer legacy code and reporting processes
• Good stakeholder engagement and communication skills