Data Engineer: Azure Databricks, Delta Lake & BI

EXASOFT CONSULTING PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

EXASOFT CONSULTING PTE. LTD. is seeking a Data Engineer to design and maintain robust data pipelines for business and analytics.

You will use Azure Data Factory and Azure Databricks to ingest, transform, and process data, while writing SQL, Python, and PySpark code for efficient data workflows. You will manage Delta Lake and Google BigQuery storage, monitor data observability, and build REST API integrations.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or related field.
• 4+ years of professional experience in Data Engineering, Data Analytics, or related technical role.
• Hands-on experience with SQL and Azure Databricks for data processing.
• Experience with PySpark, Python, and Azure Data Factory for developing data processing workflows.
• Experience with Azure DevOps, Delta Lake, REST API, JSON, and Google BigQuery.
• Experience with cloud-based data platforms and scalable data pipelines.
• Experience in Data Observability and data reliability practices.
• Experience developing Power BI dashboards for business analytics.
• Hands-on experience with JMeter for API/load/performance testing.
• Experience with ETL.
• Experience with Anomaly detection.
• Strong analytical, troubleshooting, and problem-solving skills.
• Good communication and collaboration with cross-functional teams.

Responsibilities

• Develop and maintain reliable data pipelines for business and analytical applications.
• Use Azure Data Factory and Azure Databricks to manage data ingestion, transformation, and processing workflows.
• Write optimized SQL, Python, and PySpark code for data preparation and processing.
• Work with Delta Lake and Google BigQuery for storing and managing analytical data.
• Monitor data processes and identify data-related issues using data observability practices.
• Build and maintain REST API integrations for exchanging data between applications and services.
• Process and validate JSON data used within data and API workflows.
• Support automated build and deployment activities through Azure DevOps.
• Develop Power BI dashboards and reports to present data-driven insights.
• Perform API and application performance testing using JMeter to assess response time, reliability, and system performance.
• Investigate technical issues, improve processing efficiency, and coordinate with relevant teams to resolve data-related challenges.

Skills

SQL

PySpark

Python

Power BI

Data Observability

ETL

Anomaly detection

Cross-functional collaboration

Troubleshooting

Cloud data platforms

Education

Bachelor's or Master's in Computer Science, Engineering, Data Science

Tools

Azure Databricks

Azure Data Factory

Delta Lake

Google BigQuery

REST API

JSON

Azure DevOps

JMeter

Job description

EXASOFT CONSULTING PTE. LTD. is seeking a Data Engineer to design and maintain robust data pipelines for business and analytics.

You will use Azure Data Factory and Azure Databricks to ingest, transform, and process data, while writing SQL, Python, and PySpark code for efficient data workflows. You will manage Delta Lake and Google BigQuery storage, monitor data observability, and build REST API integrations.