Databricks Data Engineer – Real-Time Pipelines & Cloud

BASIL TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Basil Technologies Pte. Ltd. is seeking a Databricks Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms.

You will integrate diverse data sources, ensure high-quality processing, and support analytics, reporting, and ML workloads. Collaboration with analytics, product, and infrastructure teams is essential to enhance the data platform while ensuring governance, monitoring, and reliability.

Qualifications

• 3+ years in data engineering with scalable pipelines
• Experience with data modeling and distributed computing architectures
• Hands-on data processing with PySpark, Spark SQL, and Databricks notebooks/jobs
• Experience orchestrating data pipelines with ADF, Airflow, or similar tools
• Experience with real-time and batch data processing
• Pipelines on Azure; AWS experience is a plus
• Proficient SQL including window functions and performance optimization
• Understanding of DevOps tools, Git workflows, and CI/CD pipelines
• Familiarity with Scrum and working in Scrum teams
• Ability to apply Scrum practices in projects
• Strong problem-solving and collaborative mindset
• Experience with streaming tech (Kafka, Flink, Kinesis) and real-time pipelines
• Ability to design and implement real-time data processing pipelines

Responsibilities

• Develop and maintain ETL pipelines for centralized data storage systems (e.g. Delta Lake)
• Integrate data from databases, APIs, logs, streaming platforms, and external providers
• Transform data to clean, normalize, and aggregate datasets
• Apply data processing techniques for complex or inconsistent data
• Contribute to frameworks and best practices for code development and deployment
• Implement data governance aligned with company standards
• Partner with analytics and product leaders to design and operationalize pipelines
• Collaborate with infrastructure leaders to advance cloud-based data platforms
• Explore new tools leveraging Azure, Databricks, or related platforms
• Monitor pipelines to detect and resolve issues promptly
• Develop monitoring tools, alerts, and automated error-handling
• Analyze business requirements and identify data extraction needs
• Attend refinement sessions with users
• Develop and maintain ETL pipelines for ingestion, transformation, validation, and loading
• Optimize performance and batch scheduling
• Develop dashboards, reports, scorecards, and data visualizations
• Perform SIT, data profiling and ensure data accuracy
• Validate completeness and consistency of ETL loads
• Support UAT and production implementation

Skills

ETL pipelines

Delta Lake

Azure

Databricks

PySpark

Spark SQL

Airflow

SQL

Data governance

CI/CD

Scrum

Kafka/Flink/Kinesis

Python

Collaboration

Tools

ADF

Git

CI/CD tools

Job description

Basil Technologies Pte. Ltd. is seeking a Databricks Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms.

You will integrate diverse data sources, ensure high-quality processing, and support analytics, reporting, and ML workloads. Collaboration with analytics, product, and infrastructure teams is essential to enhance the data platform while ensuring governance, monitoring, and reliability.