Data Engineer + AWS + Databricks

ITCAN PTE. LIMITED; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

ITCAN PTE. LIMITED in Singapore is seeking a seasoned Data Engineer to design, build and operate end-to-end data pipelines on Databricks and Snowflake, adhering to medallion architecture and governance.

You will configure workspaces, implement access controls, support migrations, and collaborate with solution architects to ensure observability and secure data handling for government/public-sector deployments.

Qualifications

• Minimum 5 years hands-on data engineering or data platform experience.
• Experience delivering data pipelines on Databricks or Snowflake.
• Strong knowledge of data governance, RBAC, and security strategies.
• Proficient in SQL and Python for large-scale transformations.

Responsibilities

• Design, build, and operate end-to-end data pipelines on Databricks/Snowflake.
• Configure compute policies, resource sizing, and access hierarchies.
• Implement data governance, lineage, masking, and security controls.
• Set up CI/CD and IaC for deployment and promotion workflows.
• Configure monitoring, telemetry, and audit logging for compliance.
• Support UAT, integration testing, and migration validation.
• Produce runbooks and access lists for operations teams.
• Collaborate with SDP Central Team and solution architects throughout delivery.

Skills

SQL

Python (PySpark)

CI/CD

Data modeling

Security/compliance

Observability

Tools

Databricks

Unity Catalog

Delta Live Tables

DABs & Repos

PySpark

SQL tooling

Terraform

Cloud platforms

Job description

KeyResponsibilities

• Design, build, and operate end-to-end data pipelines (ingestion transformation serving) on Databricks and/or Snowflake, following medallion/layered architecture patterns
• Configure and administer workspaces/accounts compute policies, resource sizing, environment setup, and access hierarchy per the SDP reference architecture
• Implement data governance controls: catalogue and schema design, RBAC, row/column-level security, data masking, and lineage tracking
• Set up CI/CD and infrastructure-as-code for pipeline deployment and environment promotion (dev test prod)
• Configure monitoring, telemetry, and audit logging to meet SDP's central observability and security posture requirements
• Support UAT, integration testing, and parallel-run validation during migration and go-live
• Produce handover documentation (runbooks, access lists, escalation procedures) for agency operations teams
• Work directly with GovTech's SDP Central Team, agency stakeholders, and Principal (Databricks/Snowflake) solution architects throughout delivery

RequiredTechnical Skills Databricks

• Unity Catalog catalogue/schema design, access control, and data lineage
• Lakeflow / Delta Live Tables for pipeline orchestration; Delta Lake table format
• Databricks SQL and cluster/workspace administration (compute policies, pools, cost management)
• Databricks Asset Bundles (DABs) and Databricks Repos for CI/CD
• PySpark / Spark SQL for large-scale data transformation
• Working knowledge of Databricks system tables (audit logs, billing/usage, query history) for observability
• Minimum 5 years of hands-on experience in Data Engineering, Data Platform Engineering, or related disciplines.
• Minimum 3 years of hands-on experience with Databricks involving data pipeline development, platform administration, governance, and optimization.

RequiredTechnical Skills Common

• Strong SQL and Python (PySpark or general-purpose) for data engineering
• Data modeling dimensional design, star/snowflake schemas, semantic layers
• CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, GitLab CI) for data engineering workflows
• Infrastructure-as-code (Terraform preferred) for provisioning cloud data platform resources
• Hands-on experience on at least one hyperscaler AWS, Azure, or Google Cloud
• Understanding of data security and compliance frameworks applicable to government/public-sector environments

Preferred Qualifications

• Databricks Certified Data Engineer Associate/Professional
• SnowPro Core, or SnowPro Advanced: Data Engineer
• Prior experience delivering on a government or regulated-sector data platform, or exposure to compliance frameworks such as IM8 is an add on
• Experience working as part of a System Integrator (SI) delivery team alongside a platform Principal is an add on