Data Engineer

Viable Solutions Pty Ltd; Sydney; On-site; AUD 120,000 - 160,000; Full time

Job summary

Viable Solutions Pty Ltd in Sydney seeks a data engineer to build and maintain dbt transformation models across pipeline layers, from raw ingestion to governed datasets.

You will write Snowflake SQL for complex transformations, develop and debug Apache Airflow DAGs on AWS ECS, and contribute to Lambda-based ingestion workflows using Python. You will also document mappings and support downstream data consumers.

Qualifications

• Proficiency with dbt Core.
• Experience with fixed-format or structured financial data files.
• Participation in code reviews and YAML-based data quality tests.
• Ownership of tickets from design through production.

Responsibilities

• Build and maintain dbt transformation models across multiple pipeline layers.
• Write Snowflake SQL for complex transformations including deduplication and surrogate keys.
• Develop and debug Apache Airflow DAGs on AWS ECS.
• Contribute to AWS Lambda-based ingestion using Python, including file validation and S3 event handling.
• Maintain source-to-target mapping documentation and align models to business data specs.
• Support onboarding of downstream consumers via data sharing mechanisms and event-driven notifications.
• Participate in code reviews and maintain data quality testing pipelines written in YAML.
• Proficiency with dbt Core; familiarity with Apache Iceberg is a plus.

Skills

dbt Core

Snowflake SQL

Apache Airflow

Python

AWS

Data quality tests

Tools

AWS MWAA

ECS

S3

Lake Formation

EventBridge

Lambda

Glue Catalogs

Iceberg

Job description

• Building and maintaining dbt transformation models across multiple pipeline layers — from raw ingestion through to governed, consumer-ready datasets
• Writing Snowflake SQL for complex transformations: type casting, deduplication, incremental load logic, surrogate key generation
• Developing and debugging Apache Airflow DAGs on AWS ECS
• Contributing to AWS Lambda-based ingestion( Python) — file validation, control file generation, S3 event handling
• Maintaining source-to-target mapping documentation and aligning models to business requirements and data specs
• Supporting onboarding of downstream consumers via data sharing mechanisms and event-driven notifications
• Participating in code reviews, writing YAML-based data quality tests, and owning tickets from design through to production
• Proficiency with dbt Core
• Apache Iceberg Table format (Nice to have)
• AWS - Working Knowledge of S3, Lambda, MWAA, ECS, Glue Catalogs, Lake Formation, Lambda, SQS, EventBridge
• Experience with fixed-format or structured financial data files