Senior Data Engineer – Databricks

OX Consultancy Pte. Ltd.; Singapore; On-site; SGD 90,000 - 120,000; Full time

Job summary

OX Consultancy Pte. Ltd. is looking for a Senior Data Engineer with extensive experience in data engineering and a strong background in using Databricks. You will be responsible for building and maintaining production data pipelines, leveraging best practices in data governance and security.

The successful candidate must have at least 8 years of experience in data engineering roles and 2-3 years of hands-on experience with Databricks. A relevant certification is mandatory.

Qualifications

• Minimum 8 years in data engineering or related roles.
• At least 2-3 years of hands-on experience with Databricks platform.
• Proven track record of refactoring legacy code to modern frameworks.

Responsibilities

• Build and maintain production data pipelines at scale.
• Work collaboratively in cross-functional teams.
• Ensure data governance and security best practices.

Skills

Data Engineering

PySpark

Databricks Platform

SQL

Python

Cloud Platforms

DevOps Practices

Data Governance

Education

Databricks Certified Data Engineer Associate

Databricks Certified Data Engineer Professional

Tools

Databricks

Git

Job description

Essential Technical Skills

• Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns
• PySpark: Proven hands‑on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization
• Databricks Platform: Practical experience with Databricks workspace, cluster management, notebooks, and job orchestration
• Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilities and integration
• Data Modelling: Experience implementing data models including dimensional modeling, data vault, or lakehouse architectures
• Delta Lake: Understanding of Delta Lake features including ACID transactions, schema evolution, and optimization techniques
• Python: Strong Python programming skills for data processing and automation

Additional Technical Skills

• SQL proficiency for data querying and transformation
• Experience with cloud platforms (Azure, AWS, or GCP)
• Understanding of data governance and security best practices
• Knowledge of streaming data processing (Structured Streaming)
• Familiarity with DevOps practices and CI/CD pipelines
• Experience with version control systems (Git)
• Understanding of data quality frameworks and testing methodologies

Professional Experience

• Minimum 8 years in data engineering or related roles
• At least 2-3 years of hands‑on experience with Databricks platform
• Proven track record of refactoring legacy code to modern frameworks
• Experience building and maintaining production data pipelines at scale
• Background working across multiple data sources and formats
• Experience in agile development environments

Required Certifications - mandatory to have at least one certification

• Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional

Additional Certifications (Preferred)

• Databricks Certified Associate Developer for Apache Spark
• Cloud platform certifications (Azure Data Engineer Associate, AWS Certified Data Analytics, or Google Cloud Professional Data Engineer)
• Relevant data engineering or big data certifications

Soft Skills

• Strong problem‑solving and analytical thinking abilities
• Excellent communication skills to explain technical concepts clearly
• Ability to work collaboratively in cross‑functional teams
• Self‑motivated with strong attention to detail
• Adaptable to changing priorities and technologies
• Client‑focused mindset with commitment to quality delivery