Lead Data Engineer

CLP Power Hong Kong Limited; Hong Kong; On-site; HKD 600,000 - 800,000; Full time

Job summary

CLP Power Hong Kong Limited is seeking a Senior Data Engineer to design and maintain scalable data platforms on Azure. This role requires at least 8 years of experience in cloud data engineering, with strong skills in Python, PySpark, and SQL. You will integrate data from various sources and optimize pipelines for performance. The position offers a 1-year contract, focusing on collaboration with data architects, AI engineers, and business stakeholders to generate actionable insights.

Qualifications

• At least 8-year experience in data engineering or analytics engineering.
• Experience with Azure Cloud Services, Databricks & Spark, SQL scripting languages.
• Proven experience building large-scale, production-grade data pipelines.

Responsibilities

• Design, build, and maintain end‑to‑end data pipelines using Azure-native and Databricks technologies.
• Develop scalable Lakehouse architectures leveraging Databricks and Delta Lake.
• Integrate data from enterprise systems, SaaS platforms, APIs, and external providers.

Skills

Python

PySpark

SQL

Azure

CI/CD

Databricks

Education

Bachelor or Masters degree in computer science or related field

Tools

Databricks Lakehouse

Azure DevOps

Terraform

Power BI

Job description

Location: Kai Tak Office

Employment period: 1-year contract (For internal application, only short-term employment contract’s terms and condition will be offered for this position regardless of job applicant’s current job category)

You will work closely with Data Architects, Platform Engineers, AI Engineers, and Business Stakeholders to design, build, and operate scalable, secure, and intelligent data platforms on Azure. You will build pipelines to gather data from across the enterprise and integrate into databases and other platforms from where it can be used by other end-users to generate meaningful insights into our business.

Reporting to the Senior Manager – Business Delivery, Data Services, the appointee will be responsible for the following duties:

Data Engineering & Platform Development

• Design, build, and maintain end‑to‑end data pipelines (batch, streaming, and CDC) using Azure‑native and Databricks technologies
• Develop scalable Lakehouse architectures leveraging Databricks, Delta Lake, and medallion design patterns (Bronze/Silver/Gold)
• Integrate data from enterprise systems, SaaS platforms, IoT sources, APIs, and external providers
• Optimize data pipelines for performance, reliability, cost, and security
• Implement data quality checks, reconciliation, and observability using automated frameworks
• Build solutions using Azure Data Factory, Azure Databricks, Azure Synapse, Azure SQL, Azure Data Lake Storage Gen2
• Implement CI/CD pipelines for data solutions using Azure DevOps / GitHub, Infrastructure as Code (Terraform)
• Apply DevOps and DataOps best practices for release management, deployment, and monitoring
• Partner with Platform and Security teams to align with cloud governance, IAM, and enterprise security standards
• Leverage AI coding agents and copilots (e.g., for Spark, SQL, Python) to:
  • Accelerate pipeline development and debugging
  • Auto‑generate transformation logic and documentation
  • Improve test coverage and code quality
  • Metadata‑driven pipelines

Requirements

Academic Qualification

• Bachelor or Masters degree in a related field (e.g., computer science, information technology, etc.)

Professional Experience

• At least 8-year experience in data engineering or analytics engineering
• Experience with Azure Cloud Services, Databricks & Spark (PySpark / Python), SQL scripting languages, relational databases (e.g., SQL DB/DW), NoSQL platforms (e.g., HBase, MongoDB, Cassandra)
• Proven experience building large‑scale, production‑grade data pipelines using on‑premises or cloud‑based data platforms
• Experience with structured, semi‑structured, and unstructured data
• Experience with Data lakes, Lakehouse, or modern data warehouse platforms
• Experience with BI and analytics integration (Power BI, Tableau, or equivalent)
• Experience in coding in data management, data warehousing or unstructured data environments
• Less experience with good attitude will be considered as Senior Data Engineer

Competencies Technical (Functional)

• Programming: Python, PySpark, SQL, Python
• Data Platforms: Databricks Lakehouse, Delta Lake
• Cloud: Azure (mandatory); AWS experience is a plus
• DevOps/DataOps: CI/CD, Git, Infrastructure as Code
• Databases: Relational (SQL DW), NoSQL (Cosmos DB, MongoDB, Cassandra)
• Streaming (nice to have): Kafka, Event Hubs, Spark Structured Streaming