Location: China
Position Overview
1. The company has established a relatively complete data warehouse and real-time data system with capabilities in data collection, computing, storage and services. With the growth of business scale and increased real-time application scenarios, we are looking to recruit two mid to senior-level data development engineers responsible for real-time data requirement delivery, task optimization and engineering governance.
2. This position focuses primarily on real-time data development, while also participating in offline data warehouse construction. You will focus on solving data latency, state expansion, resource consumption, data quality and consistency issues in the real-time data pipeline, driving the real-time data system toward greater stability, efficiency and standardization.
岗位职责
- 1. Responsible for analysis, solution design and development delivery of real-time data requirements, building real-time data processing pipelines based on technologies such as Flink and Kafka, providing data support for real-time analysis, monitoring, metrics and business decision-making scenarios.
- 2. Participate in real-time data warehouse construction and model design, complete data stratification, metric caliber, dimensional model and data service design and implementation based on business requirements.
- 3. Continuously optimize existing real-time tasks, addressing data latency, backpressure, Checkpoint exceptions, state expansion, unreasonable resource utilization and data inconsistency issues, improving task stability and computing efficiency.
- 4. Participate in data governance and engineering quality construction, including task dependency clarification, metadata management, data quality monitoring, alerting mechanisms, development standards and problem diagnosis system construction.
- 5. Participate in daily maintenance and issue handling of big data platforms and computing engines, coordinating with upstream and downstream teams to ensure stability, accuracy and timeliness of data pipelines.
- 6. Accumulate real-time development best practices and technical experience, improving the team's overall real-time R&D capability through technical sharing, code reviews and solution reviews.
Job Requirements
- 1. Bachelor's degree or above in Computer Science, Software Engineering, Information Technology or related field, with 5 years or more of data development or big data R&D experience; flexibility may be granted for exceptionally capable candidates.
- 2. Possess complete data warehouse project experience, familiar with real-time data warehouse architecture, data stratification and data model design, able to transform business requirements into data models and technical solutions.
- 3、熟练掌握Flink,具有Flink SQL、DataStream API等实际项目经验,理解State、Checkpoint、Watermark、窗口、反压和容错机制。
- 4. Possess real-time task development and optimization experience, able to independently analyze and resolve issues such as task latency, high resource consumption, state exceptions, data skewing and data consistency.
- 5. Familiar with Kafka and real-time data pipelines, understanding the complete process of data collection, transmission, computing, storage and services.
- 6. Familiar with Hadoop, Hive, Spark and other big data technologies, with certain offline data development capabilities.
- 7. Proficient in SQL, capable of completing complex data processing and performance optimization; familiar with at least one development language among Python, Java or Scala.
- 8、熟悉MySQL、PostgreSQL、Elasticsearch等常用数据库或存储系统。
- 9. Possess good business understanding, communication and collaboration skills and written expression ability, strong sense of responsibility, able to proactively drive problem resolution and project implementation.
Bonus Qualifications
- 1. Experience with large-scale Flink task development, performance tuning or cluster governance.
- 2. Experience with real-time data warehouse architecture optimization, real-time pipeline governance or technical upgrades.
- 3、熟悉Hologres、ClickHouse、Doris等实时OLAP技术。
- 4. Experience with complex business scenarios such as real-time metrics, real-time monitoring, real-time risk control or real-time recommendations.
The Candidates We Expect
- 1. We hope that you can not only complete data requirement development, but also be willing to deeply understand real-time computing mechanisms, focusing on the stability, performance and engineering quality of data pipelines.
- 2. When facing complex problems, you can identify root causes from multiple angles including data, code, task configuration and running mechanisms, and consolidate solutions into reusable standards, tools or best practices.