Databricks Data Engineer: Real-Time Pipelines & Cloud

OpenTalent; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

OpenTalent is seeking a Databricks Engineer to design, build, and maintain scalable data pipelines on Databricks and cloud platforms. You will integrate diverse data sources and support analytics, reporting, and ML workloads while aligning with governance and reliability standards.

Collaborate with analytics, product, and infrastructure teams to improve the company’s data platform, apply best practices, and stay current with streaming and real-time processing techniques across Azure and AWS

Qualifications

• 3+ years of data engineering experience with scalable pipelines.
• Experience building data solutions with data modelling and distributed architectures.
• Hands-on with PySpark, Spark SQL, and Databricks notebooks/jobs.
• Experience orchestrating pipelines with Airflow, ADF, or similar tools.
• Experience with real-time and batch processing on Azure; AWS experience beneficial.
• Strong SQL, including window functions and performance tuning.
• DevOps, Git workflows, and CI/CD familiarity.
• Familiarity with Scrum and working in Scrum teams.
• Ability to apply Scrum practices in project contexts.
• Strong problem-solving and collaborative mindset.
• Experience with streaming technologies such as Kafka, Flink, or Kinesis.
• Ability to design and implement real-time data processing pipelines.

Responsibilities

• Develop and maintain ETL pipelines for centralized data storage systems (e.g. Delta Lake).
• Integrate data from databases, APIs, log files, streaming platforms, and external providers
• Develop data transformation routines to clean, normalize, and aggregate data
• Apply data processing techniques to handle complex or inconsistent datasets
• Contribute to frameworks and best practices for code development and deployment
• Implement data governance in alignment with company standards
• Partner with analytics and product leaders to design and operationalize pipelines
• Collaborate with infrastructure leaders to advance cloud-based data platforms
• Explore new tools and techniques leveraging Azure, Databricks, or related platforms
• Monitor data pipelines to detect and resolve issues promptly
• Develop monitoring tools, alerts, and automated error-handling mechanisms
• Analyze business requirements and identify data extraction requirements
• Attend and refinement sessions with users

Skills

Data engineering

Data modelling

Distributed computing

PySpark

Spark SQL

Databricks

Airflow

Real-time processing

SQL proficiency

DevOps

Git workflows

CI/CD

Scrum

Kafka

Flink

Kinesis

Azure

AWS

Collaboration

Problem-solving

Education

Tools

Databricks

Azure

Airflow

ADF (Azure Data Factory)

Kafka

Kinesis

Flink

Delta Lake

Job description

OpenTalent is seeking a Databricks Engineer to design, build, and maintain scalable data pipelines on Databricks and cloud platforms. You will integrate diverse data sources and support analytics, reporting, and ML workloads while aligning with governance and reliability standards.

Collaborate with analytics, product, and infrastructure teams to improve the company’s data platform, apply best practices, and stay current with streaming and real-time processing techniques across Azure and AWS