Lead Data Pipeline Engineer

Recruitment Hive; City of Brisbane; On-site; AUD 120,000 - 180,000; Full time

Job summary

The Recruitment Hive invites applications for a Lead Data Pipeline Engineer based in Brisbane CBD. The role focuses on designing scalable data pipelines, optimizing performance, and collaborating with analysts and data scientists to meet project data needs.

You will automate tasks, monitor pipelines, and contribute to cloud-based data storage and processing initiatives while working within flexible day-length arrangements. This contract position expects strong GoAnywhere MFT and CI/CD experience.

Qualifications

• Minimum five years’ experience as a Data Pipeline Engineer in a Cloud Computing environment.
• Experience configuring (design, build, test, deploy) varying types of data pipelines solutions e.g. MFT, API, File extracts.
• Good command of AWS as it relates to Data Storage, Extraction, Transformation, Data processing frameworks, and security concepts around these.
• Experience with Agile development methodologies (Scrum, Kanban, Lean, Xtreme Programming).
• Minimum of 4 years in a similar role.
• Experience with GoAnywhere MFT. Experience with Ctrl-M orchestration
• Experience with CI/CD automation / GitHub / Bitbucket

Responsibilities

• Design: Creating scalable data pipelines that meet project requirements.
• Maintain: Optimising pipelines for performance and scalability.
• Monitor: Troubleshooting issues and documenting processes.
• Collaborate: Working with analysts and data scientists to understand data needs.
• Automate: Writing scripts to automate repetitive tasks.

Skills

Cloud Data Pipeline

AWS Data Services

CI/CD Automation

GoAnywhere MFT

Ctrl-M Orchestration

Agile Methodologies

Data Security

Tools

GitHub

Bitbucket

Ctrl-M

Job description

• NV1 Security Clearance - Brisbane CBD offices
• Flexible work arrangements - 7.5 hour day
• Cloud-based architecture for data storage and pipelines

Our client, an independent Federal statutory agency based in Brisbane CBD, QLD, requires the services of a Lead Data Pipeline Engineer for a 12 +12 month contract.

Duties of the Lead Data Pipeline Engineer:

• Design: Creating scalable data pipelines that meet project requirements.
• Maintain: Optimising pipelines for performance and scalability.
• Monitor: Troubleshooting issues and documenting processes.
• Collaborate: Working with analysts and data scientists to understand data needs.
• Automate: Writing scripts to automate repetitive tasks.

Skills and Experience - Lead Data Pipeline Engineer:

• Minimum of five years’ experience working as a Data Pipeline Engineer in a Cloud Computing environment.
• Demonstrated experience configuring (design, build, test, deploy) varying types of data pipelines solutions e.g. MFT, API, File extracts etc.
• Good command of AWS as it relates to Data Storage, Extraction, Transformation, Data processing frameworks, and security concepts around these.
• Experience with Agile development methodologies (Scrum, Kanban, Lean, Xtreme Programming).
• Minimum of 4 years in a similar role.
• Experience with GoAnywhere MFT. Experience with Ctrl-M orchestration
• Experience with CI/CD automation / GITHub / Bitbucket

applications close at 11am on the 16th of Sep 2026.

We Value Diversity

The Recruitment Hive is a proud Equal Opportunity Employer. We recognise the value of diverse perspectives and experience and are committed to an inclusive workplace where everyone feels valued and respected for who they are. We encourage people with disability, indigenous, veterans and people of all backgrounds and abilities to express their interest in this role.