Databricks Data Engineer - Cloud Data Pipelines

BASIL TECHNOLOGIES PTE. LTD.; Singapore; Presencial; SGD 90.000 - 180.000; Jornada completa

Descripción de la vacante

BASIL TECHNOLOGIES PTE. LTD. in Singapore is seeking a Databricks Engineer to design, build, and maintain scalable data pipelines on Databricks and cloud platforms.

You will integrate diverse data sources, ensure data quality, and support analytics, reporting, and ML workloads while partnering with analytics, product, and infrastructure teams.

The role emphasizes governance, monitoring, and reliable operations with a strong DevOps mindset.

Formación

• 3+ years of data engineering experience.
• Experience with data modelling and distributed architectures.
• Hands-on experience with PySpark, Spark SQL, and Databricks notebooks/jobs.
• Experience with orchestration tools like ADF, Airflow, or similar.
• Real-time and batch data processing experience.
• Azure and AWS cloud experience preferred.
• Proficient in SQL including window functions and optimization.
• Familiar with DevOps, Git workflows, and CI/CD pipelines.
• Experience with Scrum teams and Agile practices.
• Familiar with streaming technologies like Kafka/Flink/Kinesis.

Responsabilidades

• Develop and maintain ETL pipelines for centralized data storage (e.g., Delta Lake).
• Integrate data from databases, APIs, logs, streaming platforms, and external providers.
• Transform, clean, normalize, and aggregate data.
• Apply data processing techniques for complex datasets.
• Contribute to code development and deployment frameworks and best practices.
• Implement data governance aligned with standards.
• Collaborate with analytics, product, and infrastructure leaders to design pipelines.
• Advance cloud-based data platforms with infrastructure teams.
• Explore tools leveraging Azure, Databricks, or related platforms.
• Monitor pipelines to detect and resolve issues; build alerts and auto-recovery.
• Analyze requirements and identify data extraction needs.
• Attend and participate in refinement sessions with users.
• Develop and maintain ETL pipelines for ingestion, transformation, validation, loading.
• Optimize performance and batch scheduling.
• Develop dashboards, reports, and data visualizations.
• Perform SIT and data profiling; ensure data accuracy.
• Validate completeness and consistency of ETL loads.
• Support UAT and production implementation.

Conocimientos

Data engineering

PySpark

Spark SQL

Databricks notebooks

Azure

AWS

SQL

DevOps

Git & CI/CD

Scrum

Kafka / Kafka-like

Real-time processing

Herramientas

Databricks

Airflow

ADF

Delta Lake

Azure

AWS

Git

Descripción del empleo

BASIL TECHNOLOGIES PTE. LTD. in Singapore is seeking a Databricks Engineer to design, build, and maintain scalable data pipelines on Databricks and cloud platforms.

You will integrate diverse data sources, ensure data quality, and support analytics, reporting, and ML workloads while partnering with analytics, product, and infrastructure teams.

The role emphasizes governance, monitoring, and reliable operations with a strong DevOps mindset.