Principal Big Data Engineer

LUMINARY SERVICE PTE. LTD.; Singapore; On-site; SGD 70,000 - 110,000; Full time

Job summary

A leading technology company in Singapore seeks a Big Data Engineer to design and implement scalable data pipelines. The successful candidate will ensure data quality and integrity while integrating data from various sources. Proficiency in big data technologies and programming is essential for this role.

Qualifications

• Proven experience as a Big Data Engineer or similar role.
• Experience with Hadoop, Spark, Kafka, Hive, HBase.
• Proficiency in Python, Java, or Scala.

Responsibilities

• Design and implement scalable data pipelines for structured and unstructured data.
• Ensure data quality, integrity, and consistency across sources.
• Integrate data from databases, APIs, and third-party providers.

Skills

Data Quality Assurance

Data Integration

SQL

NoSQL

Machine Learning

Data Analytics

Python

Java

Scala

Big Data Technologies

Job description

Design and implement scalable data pipelines to collect, process, and store large volumes of structured and unstructured data.

Ensure data quality, integrity, and consistency across various data sources.

Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

Integrate data from multiple sources, including databases, APIs, and third-party data providers.

Build and maintain big data infrastructure using technologies

Optimize data storage and retrieval processes to improve performance and reduce costs.

Proven experience as a Big Data Engineer or similar role.
Experience with big data technologies such as Hadoop, Spark, Kafka, Hive, HBase, etc

Proficiency in programming languages such as Python, Java, or Scala.
Strong knowledge of SQL and NoSQL databases.
Knowledge of machine learning and data analytics.