Senior Data Engineer

Coforge; Brisbane; On-site; AUD 140,000 - 190,000; Full time

Job summary

Coforge is seeking a Senior Databricks Data Engineer in Brisbane, Australia to design, develop, and support enterprise data pipelines on AWS. The role combines platform engineering with Data BAU operations, ensuring SLA compliance, incident resolution, and continual improvement of production data ecosystems.

You will work with Unity Catalog, Delta Live Tables, serverless compute, and DBT, with experience in ITIL/ITSM practices and ticketing tools to manage incidents and changes.

Qualifications

• Strong hands-on experience in Data BAU operational support, managing production data pipelines, data freshness, and data quality alerts.
• In-depth understanding of ITIL frameworks and ITSM practices (Incident, Problem, Change, and Release management).
• Experience using ITSM ticketing tools (e.g., ServiceNow, Jira Service Management) to track SLAs, manage high priority/impact incidents, and document RCAs.
• Hands-on experience with AWS services such as S3.
• Advanced proficiency in Databricks Development work using pyspark and features: Unity Catalog, Delta Live Tables, Serverless compute, SQL Warehouse, dashboards, alerts.
• Good experience with performance tuning using Liquid Clustering, Z-Ordering, and Spark optimization.
• Practical experience managing workloads with Databricks Asset Bundles (DABs).
• Advanced experience with dbt for version-controlled data transformations.
• Kafka-based streaming data pipelines for real-time data integration.
• Proven use of Terraform for IaC to provision AWS and Databricks resources.
• Production-grade experience with GitLab for source control and CI/CD pipelines.
• Good experience with Unix/Linux environment.
• Preferred: Airline/Aviation domain experience; Databricks certifications.

Responsibilities

• Ensure Data BAU operational support for production data pipelines and data quality alerts.
• Apply ITIL/ITSM practices to Incident, Problem, Change, and Release management.
• Track SLAs and incidents using ServiceNow or Jira Service Management and document RCAs.
• Maintain and optimize Databricks workstreams and lakehouse pipelines in AWS (S3).
• Implement Unity Catalog, DLT, serverless compute, and dashboards to support analytics.
• Tune performance with Liquid Clustering, Z-Ordering, and Spark optimization.
• Manage Databricks assets with DABs and IaC via Terraform.
• Leverage dbt for transformations and Kafka-based streaming data pipelines.
• Collaborate on GitLab CI/CD pipelines and GitLab for source control.
• Work in Unix/Linux environments and ensure platform stability and SLA compliance.

Skills

Data BAU support

ITIL

ITSM

SLA management

Tools

Databricks

Unity Catalog

Delta Live Tables

Serverless compute

SQL Warehouse

Dashboarding

Liquids Clustering

Z-Ordering

Spark optimization

Databricks Asset Bundles

dbt

Kafka

Terraform

GitLab CI/CD

Unix/Linux

Job description

We are looking for a highly skilled Senior Databricks Data Engineer with strong hands-on experience in Databricks. The ideal candidate should have a solid background in designing, developing, and supporting enterprise data integration solutions, along with experience working in AWS cloud environments.

This role requires someone who will balance core platform engineering with high-stakes Data Business as Usual (BAU) operations. This engineer will have strong expertise in managing, optimizing, and supporting enterprise data pipelines hosted on AWS (S3) and Databricks. A strong background in ITIL and ITSM frameworks is critical, as you will play a key role in ensuring SLA compliance, platform stability, incident resolution, and continuous improvement for production data ecosystems.

Technical Competency & Requirements

Data BAU, ITIL & ITSM (Critical)

• Strong hands-on experience in Data BAU operational support, managing production data pipelines, data freshness, and data quality alerts.
• In-depth understanding of ITIL frameworks and ITSM practices (Incident, Problem, Change, and Release management).
• Experience using ITSM ticketing tools (e.g., ServiceNow, Jira Service Management) to track SLAs, manage high priority/impact incidents, and document RCAs.

AWS & Databricks Ecosystem

• Hands-on experience with AWS services such as S3.
• Advanced proficiency in Databricks Development work using pyspark and including contemporary features: Unity Catalog, Delta Live Tables (DLT) i.e. lakeflow pipelines, Serverless compute, SQL Warehouse, Dashboarding, Alerting etc
• Good experience working with performance tuning using Liquid Clustering, Z-Ordering, and Spark query optimization.
• Practical experience of managing workloads with Databricks Asset Bundles (DABs).

Data Ecosystem & Infrastructure

• Advanced experience with dbt (dbt Core or Cloud) for version-controlled data transformations.
• Kafka-based streaming data pipelines to support real-time and near real-time data integration requirements.
• Proven track record using Terraform for Infrastructure as Code (IaC) to provision AWS and Databricks resources.
• Production-grade experience with GitLab for source control and automated GitLab CI/CD pipelines.
• Good experience with unix/linux environment.

Preferred

• Experience in the Airline/Aviation domain is highly preferred.
• Databricks certifications will be an added advantage.