Reporting and Analytics Developer / Data Engineer 1026

USER EXPERIENCE RESEARCHERS PTE. LTD; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

USER EXPERIENCE RESEARCHERS PTE. LTD in Singapore seeks a data-focused engineer to design, build, and deploy data tables, views, and pipelines across data warehouses, data lakes, and data virtualization.

You will perform data extraction, cleaning, transformation, and data flow management, with web scraping potentially part of the scope. Collaboration with PMs, architects, and analysts will be essential to deliver scalable, data-driven products.

Qualifications

• Proficient in data cleaning and transformation to ensure data accuracy.
• Proficient in building ETL pipelines with SSIS, AWS DMS, Python, and AWS services.
• Proficient in database design across SQL, PostgreSQL, MongoDB, and MySQL.
• Experience with cloud platforms such as GCP, AWS, and Azure.
• Knowledge of data warehousing, data lakes, and data virtualization.
• Familiar with REST APIs, web protocols, and big data tools.
• Familiar with web scraping tools like BeautifulSoup, Selenium.
• Comfortable with Windows and Linux development environments.
• Interest in bridging engineering and analytics.

Responsibilities

• Design, develop, and deploy data tables, views, and marts in data warehouses, data lakes, and data virtualization.
• Perform data extraction, cleaning, transformation, and data flow management; web scraping may be part of data extraction.
• Design, build, launch, and maintain large-scale batch and real-time data pipelines.
• Integrate and collate data silos in a scalable, compliant manner.
• Collaborate with PMs, architects, BAs, frontend devs, designers, and analysts.
• Develop backend APIs and work on databases to support applications.
• Work in an Agile environment with CI/CD practices.
• Pair program and participate in code reviews with fellow developers.

Skills

Data cleaning

ETL pipelines

SQL

Python

Cloud platforms

REST APIs

Big data tools

Scripting

Windows & Linux

Collaboration/Agile

Tools

SSIS

AWS DMS

AWS Lambda

ECS

EventBridge

AWS Glue

Spring

S3

PostgreSQL

MongoDB

Job description

Key Responsibilities

• Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization.
• Perform data extraction, cleaning, transformation, and data flow management. Web scraping may also be a part of the work scope in data extraction.
• Design, build, launch, and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner that is both scalable and compliant.
• Collaborate with Project Managers, Data Architects, Business Analysts, Frontend Developers, Designers, and Data Analysts to build scalable, data-driven products.
• Be responsible for developing backend APIs and working on databases to support applications.
• Work in an Agile environment that practices Continuous Integration and Continuous Delivery.
• Work closely with fellow developers through pair programming and code review processes.

Experience and Skills Needed

• Proficient in general data cleaning and transformation (e.g., SQL, pandas, R, etc.) to ensure data accuracy and consistency.
• Proficient in building ETL pipelines (e.g., SQL Server Integration Services (SSIS), AWS Database Migration Service (DMS), Python, AWS Lambda, ECS Container Tasks, EventBridge, AWS Glue, Spring).
• Proficient in database design and various databases (e.g., SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostGIS, MySQL, SQLite, VoltDB, Cassandra, etc.).
• Experience in cloud technologies such as GCP, GCC (i.e., AWS, Azure, Google Cloud).
• Experience and passion for data engineering in a big data environment using cloud platforms such as GCP, GCC (i.e., AWS, Azure, Google Cloud).
• Experience with building production-grade data pipelines and ETL/ELT data integration.
• Knowledge of system design, data structures, and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure such as Data Marts, Data Lakes, Data Virtualization, and Data Warehouses for efficient storage and retrieval.
• Familiar with REST APIs and web requests/protocols in general.
• Familiar with big data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ).
• Familiar with W3C Document Object Model and customised web scraping (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js, etc.).
• Familiar with data governance policies, access control, and security best practices.
• Comfortable with at least one scripting language (e.g., SQL, Python).
• Comfortable working in both Windows and Linux development environments.
• Interest in being the bridge between engineering and analytics.

Bonus Experience (Added Advantage)

• Experience building data engineering pipelines that require integration with search indexes.
• Experience with Airflow and RDBMS integration and implementation (e.g., MySQL).
• Experience with either Snowflake, Databricks, or an equivalent provider