Senior Data Engineering Consultant - Pipelines & ETL

Jobtailor; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Jobtailor in Singapore is seeking an experienced Data Engineer to design and deploy data pipelines across data warehouses, data lakes, and data virtualization layers. You will perform data extraction, cleaning, and workflow orchestration, including web scraping as needed, and build scalable batch and real-time pipelines.

You will collaborate with cross-functional teams and develop backend APIs, working in an Agile CI/CD environment, with a strong emphasis on data governance, security, and

Qualifications

• Has 7 years of relevant working experience as a data engineer.
• Proficient in data cleaning and transformation (SQL, pandas, R) to ensure data accuracy.
• Proficient in building ETL/ELT pipelines (SSIS, AWS DMS, Python, AWS Lambda, ECS, EventBridge, AWS Glue, Spring).
• Proficient in database design and various databases (SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostgreSQL/GIS, MySQL, SQLite, VoltDB, Cassandra, etc).
• Experience with cloud technologies such as AWS, Azure, Google Cloud (GCC/GCP).
• Experience in big data environments using Cloud platforms.
• Experience with production-grade data pipelines, ETL/ELT integration.
• Knowledge of system design, data structures and algorithms.
• Familiar with data modelling, data access, and data storage infra like Data Mart, Data Lake, Data Virtualisation and Data Warehouse.
• Familiar with REST API and web requests/protocols.
• Familiar with Hadoop, Spark, Kafka, RabbitMQ.
• Familiar with web scraping tools (BeautifulSoup, Selenium).
• Familiar with data governance, access control and security best practices.
• Scripting in SQL/Python; comfortable on Windows and Linux.
• Interest in bridging engineering and analytics.

Responsibilities

• Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation, and flow.
• Web scraping may be also a part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• Integrate and collate data silos in a scalable and compliant manner.
• Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analysts to build scalable data-driven products.
• Be responsible for developing backend APIs & working on databases to support the applications.
• Work in an Agile Environment that practices CI/CD.
• Work closely with fellow developers through pair programming and code review process.

Skills

Data Pipeline Development

ETL Pipeline Building

Cloud Technologies

Database Design

Data Governance

Data Modeling

Big Data Tools

Web Scraping

REST API

Programming (Python)

Tools

AWS Lambda

AWS Glue

SSIS

Eventbridge

ECS Container

Data Lake

Data Mart

Data Virtualization

REST APIs

W3C DOM

Job description

Jobtailor in Singapore is seeking an experienced Data Engineer to design and deploy data pipelines across data warehouses, data lakes, and data virtualization layers. You will perform data extraction, cleaning, and workflow orchestration, including web scraping as needed, and build scalable batch and real-time pipelines.

You will collaborate with cross-functional teams and develop backend APIs, working in an Agile CI/CD environment, with a strong emphasis on data governance, security, and