Lead Data Pipeline Engineer - Cloud, AWS & Automation

Recruitment Hive; City of Brisbane; On-site; AUD 120,000 - 180,000; Full time

Job summary

The Recruitment Hive invites applications for a Lead Data Pipeline Engineer based in Brisbane CBD. The role focuses on designing scalable data pipelines, optimizing performance, and collaborating with analysts and data scientists to meet project data needs.

You will automate tasks, monitor pipelines, and contribute to cloud-based data storage and processing initiatives while working within flexible day-length arrangements. This contract position expects strong GoAnywhere MFT and CI/CD experience.

Qualifications

• Minimum five years’ experience as a Data Pipeline Engineer in a Cloud Computing environment.
• Experience configuring (design, build, test, deploy) varying types of data pipelines solutions e.g. MFT, API, File extracts.
• Good command of AWS as it relates to Data Storage, Extraction, Transformation, Data processing frameworks, and security concepts around these.
• Experience with Agile development methodologies (Scrum, Kanban, Lean, Xtreme Programming).
• Minimum of 4 years in a similar role.
• Experience with GoAnywhere MFT. Experience with Ctrl-M orchestration
• Experience with CI/CD automation / GitHub / Bitbucket

Responsibilities

• Design: Creating scalable data pipelines that meet project requirements.
• Maintain: Optimising pipelines for performance and scalability.
• Monitor: Troubleshooting issues and documenting processes.
• Collaborate: Working with analysts and data scientists to understand data needs.
• Automate: Writing scripts to automate repetitive tasks.

Skills

Cloud Data Pipeline

AWS Data Services

CI/CD Automation

GoAnywhere MFT

Ctrl-M Orchestration

Agile Methodologies

Data Security

Tools

GitHub

Bitbucket

Ctrl-M

Job description

The Recruitment Hive invites applications for a Lead Data Pipeline Engineer based in Brisbane CBD. The role focuses on designing scalable data pipelines, optimizing performance, and collaborating with analysts and data scientists to meet project data needs.

You will automate tasks, monitor pipelines, and contribute to cloud-based data storage and processing initiatives while working within flexible day-length arrangements. This contract position expects strong GoAnywhere MFT and CI/CD experience.