Databricks Data Engineer: Azure Pipelines & Real-Time Analytics

BASIL TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

Basil Technologies Pte. Ltd. in Singapore is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms.

You will integrate diverse data sources to support analytics, reporting, and machine learning, while collaborating with cross-functional teams on governance, monitoring, and reliability. The role involves implementing ETL pipelines, data transformation routines, and data quality controls, with emphasis on streaming and batch processing,

Qualifications

• 3+ years of data engineering experience with scalable pipelines.
• Experience with data modeling and distributed computing architectures.
• Hands-on PySpark, Spark SQL, and Databricks notebooks/jobs.
• Experience orchestrating pipelines with Azure Data Factory (ADF), Airflow, or similar tools.
• Experience with real-time and batch processing.
• SQL proficiency including window functions and optimization.
• Familiarity with DevOps, Git workflows, and CI/CD pipelines.

Responsibilities

• Design, develop, and maintain ETL pipelines for Delta Lake and centralized data storage.
• Integrate data from databases, APIs, log files, streaming platforms, and external providers.
• Develop data transformation routines to clean, normalize, and aggregate datasets for analysis.
• Apply data processing techniques to handle large-scale data sources.
• Contribute to frameworks and best practices for code development and deployment.
• Implement data governance policies to maintain data integrity and compliance.
• Collaborate with analytics and product teams to meet business requirements.
• Work with infrastructure teams to advance cloud-based data platforms on Azure/Databricks.
• Explore new tools to enhance data platform capabilities on Azure/Databricks.
• Monitor pipelines to detect, troubleshoot, and ensure high availability.
• Develop monitoring tools, alerts, and automated error-handling for reliability.
• Translate business requirements into data extraction and pipeline tasks.
• Participate in grooming sessions to clarify data needs.
• Optimize pipeline performance and scheduling.

Skills

3+ years data engineering

PySpark

Spark SQL

Databricks notebooks/jobs

SQL window functions

DevOps & CI/CD

Git workflows

Real-time processing

Streaming: Kafka/Flink/Kinesis

Education

Databricks Certified Data Engineer Associate

Databricks Certified Data Engineer Professional

Tools

Databricks

Azure

Azure Data Factory

Airflow

Kinesis

Job description

Basil Technologies Pte. Ltd. in Singapore is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms.

You will integrate diverse data sources to support analytics, reporting, and machine learning, while collaborating with cross-functional teams on governance, monitoring, and reliability. The role involves implementing ETL pipelines, data transformation routines, and data quality controls, with emphasis on streaming and batch processing,