Data Engineer (Databricks & DevOps)

COGNIZANT TECHNOLOGY SOLUTIONS ASIA PACIFIC PTE. LTD.; Singapore; On-site; SGD 90,000 - 150,000; Full time

Benefits offered by this job

Databricks Workflows

CloudWatch Monitoring

Job summary

Cognizant Technology Solutions Asia Pacific PTE. LTD. is seeking a Data Engineer to design, build, and maintain scalable data pipelines on AWS and Databricks. The role combines data engineering with DevOps practices to deliver automated, secure data platforms supporting analytics and ML.

Responsibilities include design of batch/streaming pipelines, lakehouse architecture, and orchestration with Airflow or Databricks Workflows. Strong Python/SQL, AWS, and CI/CD experience are essential.

Qualifications

• Bachelor's degree in CS, Engineering, or related field.
• Hands-on Databricks experience (PySpark, Delta Lake, Jobs/Workflows, cluster management).
• Strong AWS skills: S3, IAM, Glue, Lambda, EC2, Kinesis/MSK, Redshift, VPC, CloudWatch.
• Proficiency in Python and SQL.
• Experience with Git, CI/CD pipelines, Docker, and infrastructure as code (Terraform preferred).
• Understanding of data modeling, warehousing, and lakehouse concepts.
• Experience with Airflow and Databricks Workflows.
• Familiarity with Agile/Scrum delivery.

Responsibilities

• Design and build scalable data pipelines on AWS and Databricks.
• Develop batch and streaming ETL/ELT workflows.
• Implement lakehouse architectures with Delta Lake and Unity Catalog.
• Orchestrate workflows using Airflow or Databricks Workflows.
• Provision cloud infrastructure with Terraform or CloudFormation.
• Build CI/CD pipelines for data code and infra.
• Monitor pipelines and platform health (CloudWatch, Datadog).
• Optimize Spark jobs and cluster configs.
• Enforce data quality, governance, security, and access controls.
• Collaborate with data scientists, analysts, and stakeholders.
• Document pipelines, architecture, and runbooks.

Skills

Databricks PySpark

Delta Lake

AWS services

Python

SQL

CI/CD

Git

Terraform

Airflow

Spark tuning

Data governance

DevOps practices

Education

Bachelor's degree in Computer Science/Engineering

Tools

Databricks Workflows

Kubernetes

Airflow

Terraform

CloudFormation

GitHub Actions

Jenkins

Docker

Datadog

CloudWatch

Job description

We are looking for a Data Engineer to design, build, and maintain scalable data pipelines on AWS and Databricks. You will combine data engineering with DevOps practices (CI/CD, infrastructure as code, monitoring) to deliver reliable, automated, and secure data platforms that support analytics, reporting, and machine learning.

Key Responsibilities

• Design and develop batch and streaming data pipelines using Databricks (PySpark, Spark SQL, Delta Lake) on AWS
• Build and optimize ETL/ELT workflows to ingest data from databases, APIs, files, and event streams
• Implement lakehouse architecture using Delta Lake, Unity Catalog, and the medallion model (bronze/silver/gold)
• Manage and orchestrate workflows with Databricks Workflows, Airflow, or AWS Step Functions
• Provision and manage cloud infrastructure using Terraform or CloudFormation
• Build and maintain CI/CD pipelines for data code and infrastructure (GitHub Actions, GitLab CI, Jenkins, or Azure DevOps)
• Implement monitoring, logging, and alerting for pipelines and platform health (CloudWatch, Datadog, etc.)
• Optimize Spark jobs and cluster configurations for performance and cost
• Enforce data quality, governance, security, and access control standards
• Collaborate with data scientists, analysts, and business stakeholders to translate requirements into technical solutions
• Document pipelines, architecture, and operational runbooks

Required Skills and Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field
• Strong hands-on experience with Databricks (PySpark, Delta Lake, Jobs/Workflows, cluster management)
• Solid experience with AWS services: S3, IAM, Glue, Lambda, EC2, Kinesis/MSK, Redshift, VPC, CloudWatch
• Proficiency in Python and SQL
• Experience with DevOps tools and practices: Git, CI/CD pipelines, Docker, and infrastructure as code (Terraform preferred)
• Understanding of data modeling, data warehousing, and lakehouse concepts
• Experience with workflow orchestration tools (Airflow, Databricks Workflows)
• Familiarity with Agile/Scrum delivery

Preferred Qualifications

• Databricks Certified Data Engineer (Associate/Professional)
• AWS certifications (Solutions Architect, Data Engineer, or DevOps Engineer)
• Experience with Unity Catalog, DLT (Delta Live Tables), and Auto Loader
• Knowledge of Kubernetes, Kafka, or dbt
• Experience with data governance, lineage, and compliance (GDPR, PDPA, etc.)
• Exposure to MLOps or supporting ML workloads
• Scripting skills in Bash or PowerShell