Data Engineer (Databricks)

ARYAN SOLUTIONS PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

ARYAN SOLUTIONS PTE. LTD. in Singapore seeks an experienced Data Engineer to design and scale a Databricks-based data platform.

You will build robust data pipelines, implement CI/CD for data workflows, and optimize governance, security, and cost management across cloud environments. The role requires strong SQL and Python (PySpark), expertise in Unity Catalog, Delta Lake, and Delta Live Tables, plus hands-on Databricks administration.

Qualifications

• Hands-on experience with Databricks and data pipelines.
• Strong SQL and Python for data processing.
• Experience with CI/CD and IaC for data platforms.
• Knowledge of data modeling (star/snowflake) and semantic layers.
• Experience on at least one hyperscaler (AWS/Azure/GCP).
• Governance, security, and cost-management awareness.

Responsibilities

• Design and implement Databricks-based data pipelines and governance models.
• Administer Databricks environment (clusters, compute pools, policies).
• Orchestrate pipelines using Delta Live Tables and Lakeflow where applicable.
• Perform cost control, observability, and data lineage across the platform.
• Collaborate with data scientists and analysts to deliver scalable solutions.

Tools

Databricks (platform)

SQL

Python (PySpark)

Unity Catalog

Delta Lake

Delta Live Tables

CI/CD

Terraform

Cloud (AWS/Azure/GCP)

Databricks Repos

Databricks Asset Bundles

Observability tools (audit logs, billing)

Job description

Required Technical Skills — Databricks

• Unity Catalog — catalogue/schema design, access control, and data lineage
• Lakeflow / Delta Live Tables for pipeline orchestration; Delta Lake table format
• Databricks SQL and cluster/workspace administration (compute policies, pools, cost management)
• Databricks Asset Bundles (DABs) and Databricks Repos for CI/CD
• PySpark / Spark SQL for large-scale data transformation
• Working knowledge of Databricks system tables (audit logs, billing/usage, query history) for observability
• Minimum 5 years of hands-on experience in Data Engineering, Data Platform Engineering, or related disciplines.
• Minimum 3 years of hands-on experience with Databricks involving data pipeline development, platform administration, governance, and optimization.

Required Technical Skills — Common

• Strong SQL and Python (PySpark or general-purpose) for data engineering
• Data modeling — dimensional design, star/snowflake schemas, semantic layers
• CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, GitLab CI) for data engineering workflows
• Infrastructure-as-code (Terraform preferred) for provisioning cloud data platform resources
• Hands-on experience on at least one hyperscaler — AWS, Azure, or Google Cloud

Understanding of data security and compliance frameworks applicable to government/public-sector environments