Backend & Data Engineer: Spark, .NET & Python

Exigo Tech; Sydney; On-site; AUD 100,000 - 140,000; Full time

Job summary

Exigo Tech in Sydney is seeking a Software Engineer/Developer to design, build, and support scalable backend services and data-processing solutions. You will work with .NET Core, Python, PySpark, Spark, Hadoop-YARN, Trino, Delta Lake, Docker, PostgreSQL, and Linux to translate requirements into secure, maintainable production systems.

The role emphasizes performance, reliability, and platform scalability within a collaborative delivery team.

Qualifications

• Bachelor's degree or equivalent practical experience.
• Minimum 3.5 years of relevant professional experience in software development, backend engineering, data engineering, or a closely related role.
• Strong programming capability in C# and Python, with practical SQL experience.
• Hands-on experience developing .NET Core / ASP.NET Web APIs and REST-based integrations.
• Experience with PySpark and Apache Spark for distributed processing of large datasets.
• Working knowledge of Hadoop and YARN, including cluster execution concepts and resource management.

Responsibilities

• Design, develop, test, deploy, and maintain backend services using C#, .NET Core, and ASP.NET Web API.
• Build secure, reliable RESTful APIs that expose processed reporting data and integrate application and data platforms.
• Develop and optimize high-volume reporting and data-processing pipelines using Python, PySpark, and Apache Spark.
• Implement data transformation, validation, reconciliation, and error-handling controls to support accurate, dependable outputs.
• Work with Spark in standalone and cluster modes, including driver and executor behavior, resource allocation, job execution, and performance tuning.
• Contribute to scalable Hadoop-YARN architectures and support the migration or improvement of distributed workloads.
• Use Trino for distributed SQL processing and Delta Lake / Delta Parquet for efficient storage, querying, and data management.
• Package and deploy Spark, Trino, and related services using Docker; maintain consistent configuration across environments.
• Deploy, monitor, and troubleshoot workloads in production Linux environments, including job scheduling through cron and crontab.

Skills

C#

Python

SQL

C++ exposure

Education

Bachelor's degree in Computer Science, Information Technology, Software Engineering, or related discipline

Tools

Docker

Linux

PostgreSQL

PySpark

Apache Spark

Hadoop

YARN

Trino

Delta Lake

Job description

Exigo Tech in Sydney is seeking a Software Engineer/Developer to design, build, and support scalable backend services and data-processing solutions. You will work with .NET Core, Python, PySpark, Spark, Hadoop-YARN, Trino, Delta Lake, Docker, PostgreSQL, and Linux to translate requirements into secure, maintainable production systems.

The role emphasizes performance, reliability, and platform scalability within a collaborative delivery team.