Databricks Engineer

BASIL TECHNOLOGIES PTE. LTD.; Singapore; Vor Ort; SGD 90.000 - 180.000; Vollzeit

Zusammenfassung

BASIL TECHNOLOGIES PTE. LTD. in Singapore is seeking a Databricks Engineer to design, build, and maintain scalable data pipelines on Databricks and cloud platforms.

You will integrate diverse data sources, ensure data quality, and support analytics, reporting, and ML workloads while partnering with analytics, product, and infrastructure teams.

The role emphasizes governance, monitoring, and reliable operations with a strong DevOps mindset.

Qualifikationen

• 3+ years of data engineering experience.
• Experience with data modelling and distributed architectures.
• Hands-on experience with PySpark, Spark SQL, and Databricks notebooks/jobs.
• Experience with orchestration tools like ADF, Airflow, or similar.
• Real-time and batch data processing experience.
• Azure and AWS cloud experience preferred.
• Proficient in SQL including window functions and optimization.
• Familiar with DevOps, Git workflows, and CI/CD pipelines.
• Experience with Scrum teams and Agile practices.
• Familiar with streaming technologies like Kafka/Flink/Kinesis.

Aufgaben

• Develop and maintain ETL pipelines for centralized data storage (e.g., Delta Lake).
• Integrate data from databases, APIs, logs, streaming platforms, and external providers.
• Transform, clean, normalize, and aggregate data.
• Apply data processing techniques for complex datasets.
• Contribute to code development and deployment frameworks and best practices.
• Implement data governance aligned with standards.
• Collaborate with analytics, product, and infrastructure leaders to design pipelines.
• Advance cloud-based data platforms with infrastructure teams.
• Explore tools leveraging Azure, Databricks, or related platforms.
• Monitor pipelines to detect and resolve issues; build alerts and auto-recovery.
• Analyze requirements and identify data extraction needs.
• Attend and participate in refinement sessions with users.
• Develop and maintain ETL pipelines for ingestion, transformation, validation, loading.
• Optimize performance and batch scheduling.
• Develop dashboards, reports, and data visualizations.
• Perform SIT and data profiling; ensure data accuracy.
• Validate completeness and consistency of ETL loads.
• Support UAT and production implementation.

Kenntnisse

Data engineering

PySpark

Spark SQL

Databricks notebooks

Azure

AWS

SQL

DevOps

Git & CI/CD

Scrum

Kafka / Kafka-like

Real-time processing

Tools

Databricks

Airflow

ADF

Delta Lake

Azure

AWS

Git

Jobbeschreibung

JobRole Name: Databricks Engineer

Qualifications

• 3 or more years of experience in data engineering with scalable pipelines
• Strong experience designing data solutions including data modelling and distributed computing architectures
• Hands-on experience with data processing jobs using PySpark, Spark SQL, and Databricks notebooks/jobs
• Experience orchestrating data pipelines with ADF, Airflow, or similar tools
• Experience with both real-time and batch data processing
• Experience building pipelines on Azure, with AWS experience beneficial
• Proficiency in SQL including window functions and performance optimization
• Understanding of DevOps tools, Git workflows, and CI/CD pipelines
• Familiarity with Scrum methodology and experience working in Scrum teams
• Ability to apply Scrum practices in a practical project context
• Strong problem-solving and collaborative mindset
• Experience with streaming technologies such as Apache Kafka, Apache Flink, or AWS Kinesis
• Ability to design and implement real-time data processing pipelines

Certification:

Databricks Certified Data Engineer Associate and Databricks Certified Data Engineer Professional (Nice to Have NOT Mandatory)

Job Description

The Data Engineer will be responsible for designing, developing, and maintaining scalable and reliable data pipelines on Databricks and cloud platforms. The role requires integrating diverse data sources, ensuring high-quality data processing, and supporting analytics, reporting, and machine learning workloads. The role involves collaborating closely with analytics, product, and infrastructure teams to enhance the company’s data platform while adhering to best practices for governance, monitoring, and reliability.

What will you do?

• Develop and maintain ETL pipelines for centralized data storage systems (e.g. Delta Lake).
• Integrate data from databases, APIs, log files, streaming platforms, and external providers
• Develop data transformation routines to clean, normalize, and aggregate data
• Apply data processing techniques to handle complex or inconsistent datasets
• Contribute to frameworks and best practices for code development and deployment
• Implement data governance in alignment with company standards
• Partner with analytics and product leaders to design and operationalize pipelines
• Collaborate with infrastructure leaders to advance cloud-based data platforms
• Explore new tools and techniques leveraging Azure, Databricks, or related platforms
• Monitor data pipelines to detect and resolve issues promptly
• Develop monitoring tools, alerts, and automated error-handling mechanisms
• Analyze business requirements and identify data extraction requirements
• Attend and refinement sessions with users
• Develop and maintain ETL pipelines for ingestion, transformation, validation, and loading
• Optimize performance and batch scheduling
• Develop dashboards, reports, scorecards, and data visualizations
• Perform SIT, data profiling and confirm data accuracy
• Validate completeness and consistency of ETL Loads
• Support UAT and production implementation