Data Engineer: AI/ML Pipelines & Scalable Data Systems

Lendaly Pty; Sydney; On-site; AUD 90,000 - 150,000; Full time

Job summary

Lendaly Pty is seeking a data engineer to design, build, and maintain scalable data pipelines that collect, process, and store large volumes of structured and unstructured data for analysis. You will collaborate with data scientists, analysts and stakeholders to understand requirements and ensure data solutions meet business needs, while ensuring security and privacy.

The role requires a Bachelor’s degree in CS/Engineering (Master’s preferred) and strong programming in Python/Java/Scala or SQL,

Qualifications

• Bachelor’s degree in CS/Engineering; Master’s preferred.
• Strong programming in Python, Java, Scala, or SQL.
• Experience with Hadoop, Spark, and ETL development.

Responsibilities

• Design, build, and maintain scalable data pipelines to collect, process, and store large volumes of structured and unstructured data from various sources.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data solutions meet business needs.
• Implement and optimise data models, data architecture, and data warehousing solutions for efficient data retrieval and analysis.
• Monitor data systems’ performance, identify bottlenecks, and implement solutions to optimise data processing and storage.
• Ensure data security and privacy measures are in place to safeguard sensitive information and comply with relevant regulations.

Skills

Python

Java

Scala

SQL

Big data

Airflow

Data modeling

Data warehousing

Education

Bachelor's in CS/Engineering

Master’s degree preferred

Tools

Hadoop

Spark

Airflow

MongoDB

PostgreSQL

MySQL

Cassandra

Job description

Lendaly Pty is seeking a data engineer to design, build, and maintain scalable data pipelines that collect, process, and store large volumes of structured and unstructured data for analysis. You will collaborate with data scientists, analysts and stakeholders to understand requirements and ensure data solutions meet business needs, while ensuring security and privacy.

The role requires a Bachelor’s degree in CS/Engineering (Master’s preferred) and strong programming in Python/Java/Scala or SQL,