Pyspark Developer

Infosight Consulting Limited; Hong Kong; On-site; HKD 360,000 - 720,000; Full time

Job summary

Infosight Consulting Limited is seeking a data engineer in Hong Kong to develop, test, and maintain data processing pipelines using PySpark and related technologies. The role requires 1–3 years of Python experience, strong SQL, and a good understanding of Apache Spark.

Proficiency in Pandas and ETL processes, plus English and Cantonese communication, is essential; Putonghua is a plus. The candidate will contribute to data extraction, transformation, and loading from multiple sources, participate

Qualifications

• 1–3 years of experience in Python and data engineering.
• Strong knowledge of Spark architecture and components (RDDs, DataFrames).
• Proficient in SQL for data querying and manipulation.
• Experience with ETL pipelines using PySpark.
• Fluent in English and Cantonese; Putonghua is a plus.

Responsibilities

• Develop, test, and maintain data processing applications and pipelines using PySpark and related technologies.
• Perform data extraction, transformation, and loading (ETL) from multiple sources into target systems.
• Participate in code reviews, documentation, and continuous improvement of data processes.
• Troubleshoot and resolve issues in data processing and integration environments.
• Support the deployment, monitoring, and maintenance of data solutions in production.

Skills

Python

Data structures & algorithms

Pandas

SQL

Spark

PySpark

ETL

Tools

Apache Spark

PySpark

Job description

Requirements

• 1-3 years’ experience in Python, including data structures, algorithms, and libraries for data manipulation (e.g., Pandas).
• Good understanding of Apache Spark, its architecture, and components (RDDs, DataFrames, Datasets).
• Strong knowledge of SQL for data querying and manipulation.
• PExperience in ETL (Extract, Transform, Load) processes using PySpark.
• Good command of both spoken and written English & Cantonese. Ability to speak Putonghua is an advantage

Responsibilities

• Develop, test, and maintain data processing applications and pipelines using PySpark and related technologies.
• Perform data extraction, transformation, and loading (ETL) from multiple sources into target systems.
• Participate in code reviews, documentation, and continuous improvement of data processes.
• Troubleshoot and resolve issues in data processing and integration environments.
• Support the deployment, monitoring, and maintenance of data solutions in production.