Data Engineer

Lendaly Pty; Sydney; On-site; AUD 90,000 - 150,000; Full time

Job summary

Lendaly Pty is seeking a data engineer to design, build, and maintain scalable data pipelines that collect, process, and store large volumes of structured and unstructured data for analysis. You will collaborate with data scientists, analysts and stakeholders to understand requirements and ensure data solutions meet business needs, while ensuring security and privacy.

The role requires a Bachelor’s degree in CS/Engineering (Master’s preferred) and strong programming in Python/Java/Scala or SQL,

Qualifications

• Bachelor’s degree in CS/Engineering; Master’s preferred.
• Strong programming in Python, Java, Scala, or SQL.
• Experience with Hadoop, Spark, and ETL development.

Responsibilities

• Design, build, and maintain scalable data pipelines to collect, process, and store large volumes of structured and unstructured data from various sources.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data solutions meet business needs.
• Implement and optimise data models, data architecture, and data warehousing solutions for efficient data retrieval and analysis.
• Monitor data systems’ performance, identify bottlenecks, and implement solutions to optimise data processing and storage.
• Ensure data security and privacy measures are in place to safeguard sensitive information and comply with relevant regulations.

Skills

Python

Java

Scala

SQL

Big data

Airflow

Data modeling

Data warehousing

Education

Bachelor's in CS/Engineering

Master’s degree preferred

Tools

Hadoop

Spark

Airflow

MongoDB

PostgreSQL

MySQL

Cassandra

Job description

Improve upon existing systems that collect, manage and convert data for our teams to interpret in a meaningful way. Experience with building AI/ML models and rule-based engines required.

Responsibilities

• Design, build, and maintain scalable data pipelines to collect, process, and store large volumes of structured and unstructured data from various sources.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data solutions meet business needs.
• Implement and optimise data models, data architecture, and data warehousing solutions for efficient data retrieval and analysis.
• Monitor data systems’ performance, identify bottlenecks, and implement solutions to optimise data processing and storage.
• Ensure data security and privacy measures are in place to safeguard sensitive information and comply with relevant regulations.

Job requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• Strong programming skills in languages such as Python, Java, Scala, or SQL for data manipulation and analysis.
• Proficiency in working with big data technologies, such as Hadoop, Spark, or similar distributed computing frameworks.
• Experience with data modeling, data warehousing, and ETL development using tools like Apache Airflow or similar.
• Knowledge of database systems like MySQL, PostgreSQL, MongoDB, or Cassandra.