Data Engineer With Using Apache Spark, Python, Scala.

Tech Aalto; City of Melbourne; On-site; AUD 90,000 - 130,000; Part time

Job summary

Tech Aalto in Melbourne is seeking a Data Engineer to design, develop, and maintain robust data infrastructure and pipelines. You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.

The role involves building data processing pipelines with Apache Spark, Python, and Scala, developing streaming and batch processing workflows using Kafka, and deploying via Docker and Kubernetes.

Qualifications

• Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.
• Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.
• Experience with performance tuning and optimization of Spark jobs and SQL queries.
• Familiarity with stream processing frameworks like Kafka and messaging systems.
• Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.
• Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.

Responsibilities

• Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.
• Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.
• Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.
• Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3, Kubernetes.
• Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.
• Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines.
• Implement and adhere to best practices for data governance, security, and compliance.
• Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for dataworkflows.
• Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.

Skills

Apache Spark

Python

Scala

ETL

Data pipelines

SQL

Performance tuning

Team collaboration

Problem solving

Tools

Docker

Kubernetes

Kafka

NetApp S3

pytest

JUnit

Job description

Data Engineer with using Apache Spark, Python, Scala.

Location: Melbourne

Position Type: Contract

As a Data Engineer, you will be responsible for designing, developing, and maintaining robust data infrastructure and pipelines. You will work closely with cross-functional teams to understand business requirements, optimize data workflows, and ensure the reliability and performance of our data systems.

Key Responsibilities

• Design, build, and optimize data processing pipelines using Apache Spark, Python, Scala.
• Develop and maintain data ingestion and extraction processes, including streaming data pipelines using Kafka and batch processing workflows.
• Implement performance tuning techniques to optimize data processing and query performance, ensuring scalability and efficiency.
• Collaborate with DevOps teams to deploy and manage data infrastructure on NetApp S3 (very similar to AWS S3), Kubernetes.
• Containerize data applications using Docker and orchestrate deployment using Kubernetes for scalability and reliability.
• Develop and maintain unit tests using frameworks like pytest, junit, to ensure the quality and reliability of data pipelines.
• Implement and adhere to best practices for data governance, security, and compliance.
• Utilize Behavior-Driven Development (BDD) tools like Cucumber and Lettuce to write and execute test scenarios for dataworkflows.
• Stay up to date with emerging technologies and industry trends and evaluate their potential impact on our data infrastructure and processes.

Requirements

• Proven experience in designing and building scalable data pipelines using Apache Spark, Python, and Scala.
• Strong understanding of data warehousing concepts, ETL processes, and data modelling techniques.
• Experience with performance tuning and optimization of Spark jobs and SQL queries.
• Familiarity with stream processing frameworks like Kafka and messaging systems.
• Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.
• Experience with unit testing frameworks like pytest, junit, and BDD tools like Cucumber and Lettuce.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment