Senior Data Analyst

RemotePeople

Singapore

Presencial

SGD 120.000 - 180.000

Jornada completa

14 días+

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Descripción de la vacante

remotepeople is seeking a hands-on Big Data Engineer to own the real-time data ingestion system end-to-end in a collaborative, fast-paced environment. You will design and maintain streaming pipelines using Kafka, Spark, and Flink, optimize data models, and ensure low latency and high reliability for dashboards and analytics across the business.

Experience with cloud platforms (GCP/Azure) and Linux scripting is preferred, as you will work with cross-functional teams to standardize data processing

Formación

  • Bachelor's degree or above in Computer Science, Software Engineering, Data Science, or related majors.
  • 3+ years of professional big data development experience in the internet industry, with proven experience in end-to-end real-time data warehouse implementation and data ingestion system construction.
  • Expert-level knowledge of Kafka. Familiar with real-time data ingestion, partition strategy consumer mechanism, and capable of diagnosing and optimizing online risks such as message backlog, data skew, data loss and duplication; experienced in large-scale streaming pipeline stability governance.
  • Hands-on experience with Spark, Spark SQL, Flink and Flink SQL. Solid capabilities in streaming task development, resource tuning, latency and throughput optimization, and online troubleshooting.
  • Familiar with real-time data warehouse layered architecture and dimensional modeling methodologies.
  • Deep understanding of the big data ingestion system. Experienced in full-process real-time processing of multi-source data, including user behavior logs and business database data proficient in real-time collection, CDC synchronization, data cleaning, and exception handling with standardized implementation capabilities.
  • Proficient in at least one programming language among Python, Scala and Java with solid hands-on big data development capabilities. Familiar with Linux commands and Shell scripting, able to independently complete task deployment, log analysis and daily operation maintenance.
  • Familiar with mainstream relational databases and Binlog/CDC real-time synchronization principles. Experienced in business database real-time ingestion implementation. Knowledgeable in real-time data SLA governance, data quality monitoring, alert configuration and incident review mechanisms.
  • Experience with Google Cloud, Azure or other mainstream cloud big data platforms, as well as overseas team collaboration and real-time data governance experience is highly preferred.

Responsabilidades

  • Own the construction and maintenance of the company’s business data ingestion system.
  • Responsible for the full lifecycle development of real-time collection, cleaning, and warehousing of business logs, user behavioral data, and database operational data, ensuring the completeness, timeliness, and stability of end-to-end data ingestion pipelines.
  • Lead the implementation and iterative optimization of the real-time data warehouse architecture. Design multi-layer real-time data models, develop and tune streaming ETL pipelines, and support the stable output of real-time metrics, dashboards, and business monitoring data.
  • Collaborate closely with business teams, data analysts, and data product managers. Full understand business logic and statistical calibers, independently complete data requirement decomposition, logical sorting, development implementation, release verification, and continuous iteration.
  • Maintain and optimize Kafka streaming pipelines for real-time data ingestion and consumption. Troubleshoot and resolve core online issues including message backlog, data skew, data loss and duplication, and ensure high availability and stability of real-time data pipelines.
  • Develop, tune, and operate Spark real-time tasks. Continuously optimize computing resource utilization, data latency, and throughput, and improve the overall performance and stability of streaming data pipelines.
  • Participate in data standardization construction, real-time data quality governance, and pipeline redundancy optimization. Summarize and standardize data ingestion specifications and development best practices to improve the overall team’s data development efficiency.
  • Provide stable real-time data support for downstream business scenarios including BI dashboards, data services, user portrait platforms, and business effectiveness measurement, ensuring reliable online data service delivery.

Conocimientos

Kafka expertise
Spark/Flink
Programming: Python/Scala/Java
Real-time data warehousing
Linux/Shell
Cloud platforms (GCP/Azure)
Data modeling

Educación

Bachelor's degree in Computer Science / Software Engineering / Data Science

Herramientas

Kafka
Spark
Flink
SQL
Linux

Descripción del empleo


  • Own the construction and maintenance of the company’s business data ingestion system.

  • Responsible for the full lifecycle development of real-time collection, cleaning, and warehousing of business logs, user behavioral data, and database operational data, ensuring the completeness, timeliness, and stability of end-to-end data ingestion pipelines.

  • Lead the implementation and iterative optimization of the real-time data warehouse architecture. Design multi-layer real-time data models, develop and tune streaming ETL pipelines, and support the stable output of real-time metrics, dashboards, and business monitoring data.

  • Collaborate closely with business teams, data analysts, and data product managers. Full understand business logic and statistical calibers, independently complete data requirement decomposition, logical sorting, development implementation, release verification, and continuous iteration.

  • Maintain and optimize Kafka streaming pipelines for real-time data ingestion and consumption. Troubleshoot and resolve core online issues including message backlog, data skew, data loss and duplication, and ensure high availability and stability of real-time data pipelines.

  • Develop, tune, and operate Spark real-time tasks. Continuously optimize computing resource utilization, data latency, and throughput, and improve the overall performance and stability of streaming data pipelines.

  • Participate in data standardization construction, real-time data quality governance, and pipeline redundancy optimization. Summarize and standardize data ingestion specifications and development best practices to improve the overall team’s data development efficiency.

  • Provide stable real-time data support for downstream business scenarios including BI dashboards, data services, user portrait platforms, and business effectiveness measurement, ensuring reliable online data service delivery.


Job Requirements:


  • Bachelor's degree or above in Computer Science, Software Engineering, Data Science, or related majors. 3+ years of professional big data development experience in the internet industry, with proven experience in end-to-end real-time data warehouse implementation and data ingestion system construction.

  • Expert-level knowledge of Kafka. Familiar with real-time data ingestion, partition strategy consumer mechanism, and capable of diagnosing and optimizing online risks such as message backlog, data skew, data loss and duplication; experienced in large-scale streaming pipeline stability governance.

  • Hands-on experience with Spark, Spark SQL, Flink and Flink SQL. Solid capabilities is streaming task development, resource tuning, latency and throughput optimization, and online troubleshooting. Familiar with real-time data warehouse layered architecture and dimensional modeling methodologies.

  • Deep understanding of the big data ingestion system. Experienced in full-process real-time processing of multi-source data, including user behavior logs and business database data proficient in real-time collection, CDC synchronization, data cleaning, and exception handling with standardized implementation capabilities.

  • Proficient in at least one programming language among Python, Scala and Java with solid hands-on big data development capabilities. Familiar with Linux commands and Shell scripting, able to independently complete task deployment, log analysis and daily operation maintenance.

  • Familiar with mainstream relational databases and Binlog/CDC real-time synchronization principles. Experienced in business database real-time ingestion implementation. Knowledgeable in real-time data SLA governance, data quality monitoring, alert configuration and incident review mechanisms.

  • Experience with Google Cloud, Azure or other mainstream cloud big data platforms, as well as overseas team collaboration and real-time data governance experience is highly preferred.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Senior Data Engineers
Senior Data Engineers

VIDAA USA, Inc. • Singapore

Presencial
SGD 120.000 - 180.000
Senior Real-Time Data Analyst (Kafka & Spark)
Senior Real-Time Data Analyst (Kafka & Spark)

remotepeople • Singapore

Presencial
SGD 120.000 - 180.000
Data Warehouse Intern
Data Warehouse Intern

Fuku • Singapore

Presencial
SGD 13.000 - 20.000
Senior Data Engineer [20261042355]
Senior Data Engineer [20261042355]

Beyondsoft International (Singapore) Pte Ltd • Singapore

Presencial
SGD 120.000 - 180.000
Software Engineer (Apache / Pyspark / Python) - Contract Role
Software Engineer (Apache / Pyspark / Python) - Contract Role

QUESS ENGINEERING PTE. LTD. • Singapore

Presencial
SGD 120.000 - 180.000
Senior Data Engineer -
Senior Data Engineer -

jobster private ltd. • Singapore

Presencial
SGD 90.000 - 150.000
Data Engineer
Data Engineer

ZENITH INFOTECH (S) PTE LTD. • Singapore

Presencial
SGD 120.000 - 180.000
Data Engineer
Data Engineer

Merquri • Singapore

Presencial
SGD 70.000 - 110.000
Senior Data Engineer
Senior Data Engineer

Crédit Agricole CIB • Singapore

Presencial
SGD 120.000 - 180.000
Senior Data Engineer
Senior Data Engineer

optimum solutions (singapore) pte ltd • Singapore

Presencial
SGD 120.000 - 180.000