Data engineer (Azure)

FLINTEX CONSULTING PTE. LTD.; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A technology consulting firm in Singapore is seeking a Data Engineer with expertise in Azure, Pyspark, and Python. The role involves designing and developing data pipelines, managing data lakes, and establishing connections to various data sources. Candidates should have a Bachelor's degree in Computer Science or Engineering and at least 2 years of relevant experience. Strong problem-solving skills and ability to work independently are essential.

Qualifications

• 2 years of experience in Azure Data engineering, Python, Pyspark or Big Data development.
• Sound knowledge of Azure Synapse analytics for pipelines and orchestration.
• 1-2 years of experience in Visualization design with Power BI.

Responsibilities

• Design and development of Pyspark scripts and data pipelines.
• Implement and maintain datalakes.
• Establish connections to source data systems.

Skills

Pyspark

Azure Data Engineering

Python

SQL

Power BI

Data Warehouse

Education

Bachelor's Degree in Computer Science or Engineering

Tools

Azure Synapse

Azure DevOps

Job description

Overview

Data engineer (Azure) – Synapse and Pyspark, Python, Datawarehouse and Azure Data Explorer, Azure Devops

Responsibilities

• Design, review and development of Pyspark scripts. Testing, troubleshooting of data pipelines, orchestration
• Implement and maintain datalake
• Establishing connections to source data systems such as on-prem databases, IOT devices, APIs
• Managing the collected data in appropriate storage/database solutions e.g. file systems, SQL servers, Big Data platforms such as Hadoop, HANA, etc. as required by the specific project requirements.
• Design, development of relevant data pipelines using pyspark, copy data activities for batch ingestion
• Performing data integration e.g. using database table joins, or other mechanisms at an appropriate level as required by the analysis requirements of the project
• Deployment of pipeline artifacts from one environment to the other using Azure Devops
• Preferred to have Devops expertise

Skills & Experience

• Bachelor’s Degree in Computer Science or Engineering with 2 years of experience in Azure Data engineering, Python, Pyspark or Big Data development
• Sound Knowledge of Azure Synapse analytics for pipelines, orchestration, set up
• 1-2 experience in Visualization design and development with Power BI. Knowledge on row-level security, access control
• Sound experience in SQL, Datawarehouse, data marts, data ingestion with Pyspark and Python
• Expertise in developing and maintaining ETL processing pipelines in cloud-based platforms such as AWS, Azure, etc. (Azure Synapse or data factory preferred)
• Team player with good interpersonal, communication, and problem-solving skills.

Good to have

• Data catalog with Purview enabling effective metadata management, lineage tracking, and data discovery
• Candidates should demonstrate the ability to leverage Purview to ensure data governance, compliance, and efficient data exploration within Azure environments

Others

• Able to work independently on assignment according to agreed schedule without much supervision
• Own assignment and take initiative to resolve issues hinder completion of assignment Proactively reach out for help/guidance whenever required.