Data Development Engineer (Real-time Computing)

Gate

United States

Remote

USD 52,000 - 119,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Gate 正在中国招募中高级数据开发工程师,重点负责实时数据开发与离线数据仓库建设的结合。您将解决实时数据管线中的延迟、状态扩展、资源消耗与数据质量等问题,推动系统的稳定性、效率与标准化。

职位要求具备5年以上大数据开发经验,熟悉Flink、Kafka等核心技术,善于把业务需求转化为数据模型与技术方案,具备良好的沟通与协作能力,负责日常运维与技术分享,提高团队的实时研发能力。

Qualifications

  • 具备5年及以上数据开发或大数据R&D经验,能独立完成实时数据需求分析与实现。
  • 熟悉实时数据仓库架构,能够根据业务需求设计数据模型与技术方案。
  • 熟练掌握Flink、Flink SQL、DataStream API等实际项目经验,理解状态、Checkpoint和容错机制。
  • 具备实时任务开发与优化经验,能解决延迟、资源消耗、数据不一致等问题。

Responsibilities

  • 基于Flink/Kafka构建实时数据处理管线,为实时分析、监控、指标与业务决策提供数据支撑。
  • 参与实时数据仓库建设,完成数据分层、度量口径和维度模型设计。
  • 持续优化现有实时任务,提升稳定性和计算效率,处理延迟与Checkpoints等问题。
  • 参与数据治理与工程质量建设,定义任务依赖、元数据管理、数据质量监控等标准。
  • 日常运维与故障处理,协调上下游团队,确保数据管线的稳定、准确、时效。

Education

Bachelor's degree 及以上,计算机相关专业

Tools

Flink
Kafka
Hadoop
Hive
Spark
MySQL
PostgreSQL
Elasticsearch
SQL
Python/Java/Scala

Job description

Location: China


Position Overview

1. The company has established a relatively complete data warehouse and real-time data system with capabilities in data collection, computing, storage and services. With the growth of business scale and increased real-time application scenarios, we are looking to recruit two mid to senior-level data development engineers responsible for real-time data requirement delivery, task optimization and engineering governance.


2. This position focuses primarily on real-time data development, while also participating in offline data warehouse construction. You will focus on solving data latency, state expansion, resource consumption, data quality and consistency issues in the real-time data pipeline, driving the real-time data system toward greater stability, efficiency and standardization.


岗位职责


  • 1. Responsible for analysis, solution design and development delivery of real-time data requirements, building real-time data processing pipelines based on technologies such as Flink and Kafka, providing data support for real-time analysis, monitoring, metrics and business decision-making scenarios.

  • 2. Participate in real-time data warehouse construction and model design, complete data stratification, metric caliber, dimensional model and data service design and implementation based on business requirements.

  • 3. Continuously optimize existing real-time tasks, addressing data latency, backpressure, Checkpoint exceptions, state expansion, unreasonable resource utilization and data inconsistency issues, improving task stability and computing efficiency.

  • 4. Participate in data governance and engineering quality construction, including task dependency clarification, metadata management, data quality monitoring, alerting mechanisms, development standards and problem diagnosis system construction.

  • 5. Participate in daily maintenance and issue handling of big data platforms and computing engines, coordinating with upstream and downstream teams to ensure stability, accuracy and timeliness of data pipelines.

  • 6. Accumulate real-time development best practices and technical experience, improving the team's overall real-time R&D capability through technical sharing, code reviews and solution reviews.


Job Requirements


  • 1. Bachelor's degree or above in Computer Science, Software Engineering, Information Technology or related field, with 5 years or more of data development or big data R&D experience; flexibility may be granted for exceptionally capable candidates.

  • 2. Possess complete data warehouse project experience, familiar with real-time data warehouse architecture, data stratification and data model design, able to transform business requirements into data models and technical solutions.

  • 3、熟练掌握Flink,具有Flink SQL、DataStream API等实际项目经验,理解State、Checkpoint、Watermark、窗口、反压和容错机制。

  • 4. Possess real-time task development and optimization experience, able to independently analyze and resolve issues such as task latency, high resource consumption, state exceptions, data skewing and data consistency.

  • 5. Familiar with Kafka and real-time data pipelines, understanding the complete process of data collection, transmission, computing, storage and services.

  • 6. Familiar with Hadoop, Hive, Spark and other big data technologies, with certain offline data development capabilities.

  • 7. Proficient in SQL, capable of completing complex data processing and performance optimization; familiar with at least one development language among Python, Java or Scala.

  • 8、熟悉MySQL、PostgreSQL、Elasticsearch等常用数据库或存储系统。

  • 9. Possess good business understanding, communication and collaboration skills and written expression ability, strong sense of responsibility, able to proactively drive problem resolution and project implementation.


Bonus Qualifications


  • 1. Experience with large-scale Flink task development, performance tuning or cluster governance.

  • 2. Experience with real-time data warehouse architecture optimization, real-time pipeline governance or technical upgrades.

  • 3、熟悉Hologres、ClickHouse、Doris等实时OLAP技术。

  • 4. Experience with complex business scenarios such as real-time metrics, real-time monitoring, real-time risk control or real-time recommendations.


The Candidates We Expect


  • 1. We hope that you can not only complete data requirement development, but also be willing to deeply understand real-time computing mechanisms, focusing on the stability, performance and engineering quality of data pipelines.

  • 2. When facing complex problems, you can identify root causes from multiple angles including data, code, task configuration and running mechanisms, and consolidate solutions into reusable standards, tools or best practices.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Real-Time Data Streaming Engineering Lead
Real-Time Data Streaming Engineering Lead

Weekday • United States

On-site
USD 160,000 - 230,000
Staff Software Engineer – Data
Staff Software Engineer – Data

Sony Interactive Entertainment (SIE) • United States

Remote
USD 180,000 - 290,000
Health insurance
401(k) matching
Paid time off
+3
Data Development Engineer (Real-time Computing)
Data Development Engineer (Real-time Computing)

Gate.io • United States

Remote
USD 90,000 - 130,000
Flink Leader
Flink Leader

Weekday AI • United States

On-site
USD 180,000 - 240,000
Real-Time Data Engineer – Spark, Kafka & GCP
Real-Time Data Engineer – Spark, Kafka & GCP

Compunnel Inc. • Sunnyvale (CA)

On-site
USD 100,000 - 140,000
Data & Software Engineer — Real‑Time, Scalable Pipelines
Data & Software Engineer — Real‑Time, Scalable Pipelines

Roberts Recruiting, LLC • Boston (MA)

On-site
Confidential
Competitive comp based on experience
Healthcare HMO & PPO
Stock options and 401k
+1
Big Data Engineer
Big Data Engineer

Princeton IT Services, Inc • Mount Laurel Township (NJ)

On-site
USD 120,000 - 150,000
Senior Flink Architect & Team Lead
Senior Flink Architect & Team Lead

Weekday • United States

On-site
USD 180,000 - 240,000
Senior Data Engineer
Senior Data Engineer

CIS Technologies INC • United States

Remote
USD 120,000 - 160,000
Data Engineer — Real-Time FinTech Pipelines
Data Engineer — Real-Time FinTech Pipelines

Vorkis Llc • Texas

Remote
USD 90,000 - 120,000