Data Engineer: ETL on Databricks & AWS

CITCO INTERNATIONAL SUPPORT SERVICES LIMITED-PHILIPPINE ROHQ; Santol; On-site; PHP 900,000 - 1,300,000; Full time

Benefits offered by this job

Benefit program

Job summary

Citco is seeking an IT Developer - Data Engineer to design, implement, and optimize ETL/ELT pipelines using Databricks on AWS. You will migrate data platforms from on-premise or other clouds, ensuring data integrity and minimal disruption during transitions.

You will work with Databricks notebooks, jobs, and AWS Glue workflows to create scalable data models for batch and streaming workloads, while upholding security and governance standards and exploring AI/ML opportunities.

Qualifications

• 3-5 years of hands-on experience in data engineering, with at least 2-3 years on Databricks including notebooks, clusters, jobs, Delta Live Tables, and Unity Catalog.
• Strong hands-on experience with Databricks core components: Delta Lake, Delta Live Tables, Databricks Workflows, Unity Catalog, Data Governance, AutoLoader, Structured Streaming.
• Proficient in SQL, Python, PySpark, and/or Scala within the Databricks environment; strong AWS integration (S3, Glue, Athena, Lambda).
• Hands-on experience building and managing ETL/ELT jobs in AWS (Glue Jobs, Crawlers, Data Catalog); real-time and batch processing with Databricks.
• Familiar with PL/SQL and SQL scripting; BI tools like Power BI and/or Qlik.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Databricks Delta Lake and Apache Spark.
• Support data platform migration activities to Databricks on AWS; migrate pipelines, workflows, and data assets.
• Build and manage AWS-based data pipelines using AWS Glue, S3, and related ETL services.
• Implement data workflows using Databricks notebooks, jobs, and AWS Glue workflows.
• Create and optimize data models for batch and streaming data processing.
• Develop data quality frameworks and monitoring during and after migration.
• Collaborate with cross-functional teams to meet data requirements with scalable solutions.
• Write efficient SQL queries and optimize performance.
• Implement data security and governance aligned with standards.
• Maintain documentation for data processes, runbooks, and architectures (e.g., Medallion Architecture).
• Explore Gen AI / ML capabilities on data pipelines where applicable.
• Stay current with Databricks, AWS, and emerging data tools.

Tools

Delta Lake

Delta Live Tables

Databricks Workflows

Unity Catalog

Data Governance

Databricks AutoLoader

Structured Streaming

SQL

Python

PySpark

Scala

S3

AWS Glue

Athena

Lambda

Glue Jobs

Glue Crawlers

Data Catalog

ETL/ELT

Power BI

Qlik

PL/SQL

Job description

Citco is seeking an IT Developer - Data Engineer to design, implement, and optimize ETL/ELT pipelines using Databricks on AWS. You will migrate data platforms from on-premise or other clouds, ensuring data integrity and minimal disruption during transitions.

You will work with Databricks notebooks, jobs, and AWS Glue workflows to create scalable data models for batch and streaming workloads, while upholding security and governance standards and exploring AI/ML opportunities.