Databricks Data Engineer

RIDIK Pte Ltd; Singapore; On-site; SGD 140,000 - 190,000; Full time

Job summary

RIDIK Pte Ltd is seeking a senior data engineer to design and run scalable data pipelines on the Databricks platform in Singapore. You will implement end-to-end workflows, optimize Spark jobs, and collaborate with data teams to deliver reliable data products.

The role requires strong PySpark, Delta Lake, and Python skills, with experience migrating ETL to ELT and working in agile environments. On-site Singapore location with collaborative teams.

Qualifications

• Experience in data engineering, ETL/ELT patterns, and scalable pipelines.
• Hands-on PySpark development with DataFrames, Spark SQL, and performance tuning.
• Familiarity with Databricks workspace, notebook usage, and job orchestration.
• Knowledge of Databricks Workspace AI Agent capabilities and integration.
• Experience with data modelling (dimensional modeling, data vault, lakehouse) and Delta Lake features.
• Strong Python for data processing and automation.

Responsibilities

• Design, build, and operate scalable data pipelines on Databricks.
• Develop end-to-end workflows from ingestion to consumption with robust error handling.
• Monitor, alert, and maintain data pipelines for reliability and performance.
• Refactor legacy pipelines to PySpark and migrate ETL to ELT on Databricks.
• Collaborate with data architects, analysts, and stakeholders; mentor junior engineers.

Skills

Data Engineering

PySpark

Databricks Platform

Workspace AI Agent

Data Modelling

Delta Lake

Python

Tools

SQL

Cloud Platforms (Azure/AWS/GCP)

Structured Streaming

CI/CD

Git

Data Governance & Security

Job description

Essential Technical Skills

• Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns
• PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization
• Databricks Platform: Practical experience with Databricks workspace, cluster management, notebooks, and job orchestration
• Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilities and integration
• Data Modelling: Experience implementing data models including dimensional modeling, data vault, or lakehouse architectures
• Delta Lake: Understanding of Delta Lake features including ACID transactions, schema evolution, and optimization techniques
• Python: Strong Python programming skills for data processing and automation

Additional Technical Skills

• SQL proficiency for data querying and transformation
• Experience with cloud platforms (Azure, AWS, or GCP)
• Understanding of data governance and security best practices
• Knowledge of streaming data processing (Structured Streaming)
• Familiarity with DevOps practices and CI/CD pipelines
• Experience with version control systems (Git)
• Understanding of data quality frameworks and testing methodologies

Professional Experience

• Minimum 8 years in data engineering or related roles
• At least 2-3 years of hands-on experience with Databricks platform
• Proven track record of refactoring legacy code to modern frameworks
• Experience building and maintaining production data pipelines at scale
• Background working across multiple data sources and formats
• Experience in agile development environments

Required Certifications

• Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional

Data Pipeline Development & Operations

• Design, build, and operate scalable and reliable data pipelines on the Databricks platform
• Develop end-to-end data workflows from ingestion through transformation to consumption
• Implement robust error handling, monitoring, and alerting mechanisms
• Ensure data pipeline reliability, performance, and maintainability
• Optimize pipeline performance through efficient Spark job design and cluster configuration
• Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization

• Refactor legacy code and data pipelines to PySpark for improved performance and scalability
• Migrate traditional ETL processes to modern ELT patterns on Databricks
• Assess existing codebases and identify opportunities for optimization and modernization
• Ensure backward compatibility and data integrity during migration processes
• Document refactoring approaches and create migration playbooks
• Collaborate with stakeholders to minimize disruption during code transitions

RESPONSIBILITIES

Data Engineering Excellence

• Implement data quality checks and validation frameworks
• Design and maintain Delta Lake tables with appropriate optimization strategies
• Develop reusable code libraries and frameworks for common data engineering tasks
• Follow software engineering best practices including version control, testing, and CI/CD
• Participate in code reviews and provide constructive feedback to team members
• Troubleshoot and resolve data pipeline issues in production environments

Collaboration & Knowledge Sharing

• Work closely with data architects, analysts, and business stakeholders
• Collaborate with Infrastructure (Infra), Applications (Apps), and Cyber teams
• Share knowledge and best practices with Team NCS
• Mentor junior data engineers on PySpark and Databricks technologies
• Document technical solutions and maintain comprehensive documentation