Data Engineer (Senior Consultant)

HCL Technologies Limited; Singapore; Vor Ort; SGD 110.000 - 150.000; Vollzeit

Zusammenfassung

HCL Technologies Limited in Singapore is seeking a Data Engineer to design, build, and maintain data infrastructure supporting statistical and analytical production processes. You will lead in developing reliable data pipelines, validate data, and ensure outputs reconcile with production results, while guiding solution design and engineering standards.

Collaborating with statisticians and business stakeholders, you will modernize legacy processes, implement scalable batch and real-time

Qualifikationen

• 7+ years in data engineering or related software engineering roles with production-grade data solutions.
• Strong Python and SQL skills for complex data transformations, analysis, and quality monitoring.
• Experience with batch processing, ETL, data integration, and data pipeline architecture.
• Proficiency with Linux/Unix, shells scripting, and command-line tools for data processing.
• Knowledge of software engineering practices: modular design, logging, testing, and documentation.
• Experience with cloud platforms and orchestration/CI-CD tooling is preferred.
• Ability to lead or mentor data engineers and provide technical guidance.

Aufgaben

• Lead the architecture, design, and implementation of ETL processes and data pipelines from multiple sources.
• Build and maintain large-scale batch and real-time data pipelines with proper monitoring and failure handling.
• Develop production-quality code for statistical production processes, modernising legacy pipelines while preserving logic.
• Lead database/data-model design and performance optimisations for large datasets.
• Promote reusable data engineering components and standardised configurations across the team.
• Design CI/CD for data workflows and manage development, UAT, and production environments.
• Oversee complex pipeline workflows and scheduling using enterprise tools.
• Establish version control structures (GitLab) and code review practices.
• Troubleshoot production incidents and continuously optimise pipeline performance.
• Provide technical mentoring to data engineers and guidance on best practices.
• Collaborate with business users to translate requirements into robust technical solutions.
• Identify risks and opportunities and recommend design or implementation approaches.

Kenntnisse

Python

SQL

ETL

Data pipelines

Linux/Unix

Git

CI/CD

Cloud platforms

Tools

GitLab

Jobbeschreibung

The Data Engineer works closely with statisticians, analytical teams, and business stakeholders to design, build, and maintain the data infrastructure needed to support rigorous statistical production processes. The role centres on building reliable and maintainable data pipelines, validation and frameworks that support statistical production processes, while providing technical leadership across the engineering team.

You will work closely with the user team on the business needs, translate them into robust technical solutions and lead in the modernisation of legacy processes. This includes ensuring that existing business logic is preserved and that outputs are accurately reconciled against current production results. In addition, you will provide technical guidance on solution design, engineering standards, and implementation approaches to ensure the team delivers production-quality solutions.

Key Responsibilities

• Leads the architecture, design, and implementation of ETL processes, workflows, and data pipelines for the ingestion, transformation, and integration of data from multiple source agencies.
• Build and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks, including dependency management, failure handling, recovery, and operational monitoring.
• Develop and maintain modular, production-quality code for statistical production processes, including uplifting of legacy processes while preserving business logic and reconciling outputs against existing results. Review complex business logic and provide technical direction for migration and modernisation activities.
• Lead database and data model design, including schema design, storage patterns and performance optimisation for large-scale statistical datasets.
• Design and drive the adoption of reusable data engineering components such as common libraries, configuration frameworks, logging standards, and utilities to streamline production workflows and promote consistent engineering practices across the team.
• Design and implement CI/CD for data engineering workflows to automate testing and deployment, and establish workflows to safely isolate and synchronise Development, UAT, and Production environments.
• Lead the design and configuration of complex pipeline workflows and schedules using enterprise tools, including dependency management, failure handling, recovery and operational monitoring.
• Establish version control structures using GitLab, set up code review frameworks, and document technical architectures. Provide technical reviews for solution designs, merge requests and implementation approaches, and promote appropriate engineering standards across the development team.
• Troubleshoot production incidents, monitor pipeline health, and continuously optimise system performance for large-scale datasets. Lead the investigation of complex production issues, identify root causes, and implement preventive and performance improvement measures.
• Lead and provide technical guidance and mentoring to data engineers, including support on coding practices, solution design, troubleshooting and implementation decisions.
• Collaborate with technical team, business users and other stakeholders to translate complex business and statistical processing requirements into robust technical solutions.
• Identify technical risks, dependencies and improvement opportunities, and recommend appropriate design or implementation approaches.

Required Skills & Experience

• At least 7 years of professional experience in data engineering, data platform development, or a related software engineering role, with demonstrated experience designing and implementing production-grade data solutions at scale.
• Strong proficiency in Python and SQL, including experience in complex data transformations, analysis, and quality monitoring, with the ability to design, review and optimise complex processing logic.
• Strong experience with batch processing, ETL, data integration and data pipeline architecture.
• Strong working knowledge of Linux/Unix environments, shell scripting and command-line tools for data processing
• Strong understanding and practical application of software engineering practices such as modular design, configuration management, logging, exception handling, testing, and technical documentation, with experience defining or applying engineering standards for production systems.
• Experience with designing technical solutions and making engineering trade-off decisions involving performance, maintainability, scalability and operational reliability.
• Experience with leading or providing technical guidance for data pipeline development, system migration or modernisation initiatives.
• Experience with conducting code reviews and mentoring or providing technical guidance.
• Experience with version control using Git, including branching strategies, merge requests, code review, and CI/CD pipelines for automated testing and deployment
• Experience with cloud platforms and orchestration/CI-CD tooling is preferred.
• Excellent problem-solving, analytical, and communication skills, with the ability to work independently in an agile team environment while providing technical leadership when resolving complex engineering problems.