Data Platform Engineer: End-to-End Pipelines & Governance

Eightcap; City of Melbourne; On-site; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Wellness massages

Drinks every Friday

Parental leave

Staff referral bonus

Flu vaccinations

Purchased leave

Novated leasing

CBD location

Collaborative culture

Social activities

Job summary

Eightcap is seeking a Data Engineer to own end-to-end data pipelines and influence the data platform's growth. You will design and maintain pipelines that ingest internal, API, and third-party data into a scalable data lake, while ensuring quality, security and governance.

You will write robust SQL and Python, utilize PySpark, dbt, AWS Glue/EMR, and manage costs and reliability. Collaboration with teams and mentorship are key parts of the role.

Qualifications

• 3+ years experience as a Data Engineer with ownership of significant pipelines.
• Experience designing end-to-end data pipelines and data platforms.

Responsibilities

• Design, build and maintain end-to-end pipelines feeding data into a data lake.
• Contribute to data lake/platform architecture and documentation.
• Write high-quality SQL and Python code, leveraging PySpark, dbt, and AWS tools.
• Implement data quality, governance, and compliance controls (ASIC reporting).
• Monitor, troubleshoot and optimize pipelines for performance and cost.
• Mentor junior engineers and lead technical projects.
• Evaluate new data sources and maintain modern tooling.

Skills

SQL

Python

PySpark

Cloud data lake

ETL/ELT

AWS Glue

AWS EMR

S3

Redshift

Data governance

CI/CD

DevOps

IAM

VPC

Education

Bachelor's degree in Computer Science, Software Engineering, or related field

Tools

PySpark

dbt

AWS Glue

Emr

S3

X

Job description

Eightcap is seeking a Data Engineer to own end-to-end data pipelines and influence the data platform's growth. You will design and maintain pipelines that ingest internal, API, and third-party data into a scalable data lake, while ensuring quality, security and governance.

You will write robust SQL and Python, utilize PySpark, dbt, AWS Glue/EMR, and manage costs and reliability. Collaboration with teams and mentorship are key parts of the role.