Data Engineer (SQL, Azure Databricks, PySpark, Azure Data Factory, Python, Azure DevOps, Power BI, Delta Lake, ETL)

NOVACLOUD SYSTEMS PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

NOVACLOUD SYSTEMS PTE. LTD. is seeking a data engineer to build and maintain reliable data pipelines using Azure Data Factory, Azure Databricks, Delta Lake, and BigQuery.

You will write SQL, Python, and PySpark to process data and develop scalable workflows, with a focus on observability and reliability across cloud data platforms. You will implement REST API integrations, validate JSON data, and create Power BI dashboards while supporting CI/CD through Azure DevOps and performing API testing

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related technical discipline.
• 4+ years of professional experience in Data Engineering, Data Analytics, or a related technical role.
• Strong hands-on experience with SQL and Azure Databricks for data processing and analytical workloads.
• Strong experience in PySpark, Python, and Azure Data Factory for developing and managing data processing workflows.
• Experience in practical knowledge of Azure DevOps, Delta Lake, REST API, JSON, and Google BigQuery.
• Experience in working with cloud-based data platforms and developing scalable data pipelines.
• Strong experience of Data Observability, data monitoring, and data reliability practices.
• Experience in developing Power BI dashboards and reports for business and analytical requirements.
• Hands-on experience in JMeter for API, load, or performance testing.
• Hands-on experience in ETL.
• Experience in Anomaly.
• Strong analytical, troubleshooting, and problem-solving skills.
• Good communication and collaboration skills with the ability to work effectively with cross-functional teams.

Responsibilities

• Develop and maintain reliable data pipelines for business and analytical applications.
• Use Azure Data Factory and Azure Databricks to manage data ingestion, transformation, and processing workflows.
• Write optimized SQL, Python, and PySpark code for data preparation and processing.
• Work with Delta Lake and Google BigQuery for storing and managing analytical data.
• Monitor data processes and identify data-related issues using data observability practices.
• Build and maintain REST API integrations for exchanging data between applications and services.
• Process and validate JSON data used within data and API workflows.
• Support automated build and deployment activities through Azure DevOps.
• Develop Power BI dashboards and reports to present data-driven insights.
• Perform API and application performance testing using JMeter to assess response time, reliability, and system performance.
• Investigate technical issues, improve processing efficiency, and coordinate with relevant teams to resolve data-related challenges.

Skills

SQL

Azure Databricks

PySpark

Python

Azure Data Factory

Delta Lake

REST API

JSON

Google BigQuery

Power BI

JMeter

ETL

Data Observability

Data Reliability

Education

Bachelor's or Master's in CS, Engineering, Data Science

Tools

Azure DevOps

Delta Lake

REST API

Job description

Responsibilities:

• Develop and maintain reliable data pipelines for business and analytical applications.
• Use Azure Data Factory and Azure Databricks to manage data ingestion, transformation, and processing workflows.
• Write optimized SQL, Python, and PySpark code for data preparation and processing.
• Work with Delta Lake and Google BigQuery for storing and managing analytical data.
• Monitor data processes and identify data-related issues using data observability practices.
• Build and maintain REST API integrations for exchanging data between applications and services.
• Process and validate JSON data used within data and API workflows.
• Support automated build and deployment activities through Azure DevOps.
• Develop Power BI dashboards and reports to present data-driven insights.
• Perform API and application performance testing using JMeter to assess response time, reliability, and system performance.
• Investigate technical issues, improve processing efficiency, and coordinate with relevant teams to resolve data-related challenges.

Requirements:

• Qualification in Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related technical discipline.
• 4+ years of professional experience in Data Engineering, Data Analytics, or a related technical role.
• Strong Hands - on experience with SQL and Azure Databricks for data processing and analytical workloads.
• Strong Experience in PySpark, Python, and Azure Data Factory for developing and managing data processing workflows.
• Experience in practical knowledge of Azure DevOps, Delta Lake, REST API, JSON, and Google BigQuery.
• Experience in working with cloud-based data platforms and developing scalable data pipelines.
• Strong Experience of Data Observability, data monitoring, and data reliability practices.
• Experience in developing Power BI dashboards and reports for business and analytical requirements.
• Hands - on experience in JMeter for API, load, or performance testing.
• Hands - on experience in ETL.
• Experience in Anomaly.
• Strong analytical, troubleshooting, and problem-solving skills.
• Good communication and collaboration skills with the ability to work effectively with cross-functional teams.