Databricks Data Engineer

COGNIZANT TECHNOLOGY SOLUTIONS ASIA PACIFIC PTE. LTD.; Singapore; On-site; SGD 90,000 - 170,000; Full time

Job summary

Cognizant Technology Solutions Asia Pacific PTE. LTD. seeks a senior data engineer to design, build, and operate end-to-end data pipelines on Databricks and Snowflake, following medallion patterns.

You will configure workspaces, implement data governance, and set up CI/CD and IaC for deployment, while collaborating with client stakeholders and platform architects to ensure compliant, scalable data delivery across government/regulated environments.

Qualifications

• Strong SQL and Python (PySpark) for data engineering.
• Data modeling — dimensional design, star/snowflake schemas.
• CI/CD pipelines for data engineering workflows.
• Infrastructure‑as‑code (Terraform preferred) for provisioning cloud data resources.
• Hands‑on experience on at least one hyperscaler (AWS/Azure/GCP).
• Understanding of data security/compliance for government/public‑sector environments.

Responsibilities

• Design, build, and operate end‑to‑end data pipelines on Databricks/Snowflake using medallion patterns.
• Configure and administer workspaces/accounts per SDP reference architecture.
• Implement data governance: catalogu e/schema design, RBAC, data masking, lineage.
• Set up CI/CD and IaC for pipeline deployment (dev → test → prod).
• Configure monitoring, telemetry, and audit logging for observability and security.
• Support UAT, integration testing, and parallel‑run validation during migration/go‑live.
• Produce handover documentation for agency operations teams.
• Collaborate with client, agency stakeholders, and Principal solution architects.

Skills

Strong SQL

Python (PySpark)

Data modeling

CI/CD pipelines

Infrastructure‑as‑code

Hyperscaler knowledge

Data security/compliance

Tools

Databricks Unity Catalog

Delta Live Tables

Delta Lake

Databricks SQL

Databricks Repos

DABs

Job description

Key Responsibilities

• Design, build, and operate end-to-end data pipelines (ingestion → transformation → serving) on Databricks and/or Snowflake, following medallion/layered architecture patterns
• Configure and administer workspaces/accounts — compute policies, resource sizing, environment setup, and access hierarchy — per the SDP reference architecture
• Implement data governance controls: catalogue and schema design, RBAC, row/column-level security, data masking, and lineage tracking
• Set up CI/CD and infrastructure-as-code for pipeline deployment and environment promotion (dev → test → prod)
• Configure monitoring, telemetry, and audit logging to meet SDP's central observability and security posture requirements
• Support UAT, integration testing, and parallel-run validation during migration and go-live
• Produce handover documentation (runbooks, access lists, escalation procedures) for agency operations teams
• Work directly with client, agency stakeholders, and Principal (Databricks/Snowflake) solution architects throughout delivery

Required Technical Skills — Databricks

• Unity Catalog — catalogue/schema design, access control, and data lineage
• Lakeflow / Delta Live Tables for pipeline orchestration; Delta Lake table format
• Databricks SQL and cluster/workspace administration (compute policies, pools, cost management)
• Databricks Asset Bundles (DABs) and Databricks Repos for CI/CD
• PySpark / Spark SQL for large-scale data transformation
• Working knowledge of Databricks system tables (audit logs, billing/usage, query history) for observability
• Minimum 5 years of hands‑on experience in Data Engineering, Data Platform Engineering, or related disciplines.
• Minimum 3 years of hands‑on experience with Databricks involving data pipeline development, platform administration, governance, and optimization.

Required Technical Skills

• Strong SQL and Python (PySpark or general-purpose) for data engineering
• Data modeling — dimensional design, star/snowflake schemas, semantic layers
• CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, GitLab CI) for data engineering workflows
• Infrastructure-as-code (Terraform preferred) for provisioning cloud data platform resources
• Hands‑on experience on at least one hyperscaler — AWS, Azure, or Google Cloud
• Understanding of data security and compliance frameworks applicable to government/public‑sector environments

Preferred Qualifications

• Databricks Certified Data Engineer Associate/Professional
• SnowPro Core, or SnowPro Advanced: Data Engineer
• Prior experience delivering on a government or regulated‑sector data platform, or exposure to compliance frameworks such as IM8 is an add on
• Experience working as part of a System Integrator (SI) delivery team alongside a platform Principal is an add on