Senior Data Engineer: Scalable AI-Driven Data Pipelines

Datasmiths; Australia; Hybrid; AUD 140,000 - 200,000; Full time

Benefits offered by this job

Hybrid and remote work options

Competitive compensation

Flexible work culture

Growth and learning resources

Job summary

Datasmiths in Australia is seeking a Senior Data Engineer to design and optimize scalable data architectures and big data pipelines. You will work with cloud-native solutions, Databricks, Spark, and Delta Lake to deliver AI-powered insights.

The role emphasizes automation, governance, and collaboration with data scientists, analysts, and engineers to drive data-driven outcomes. Hybrid and remote options are available.

Qualifications

• 5+ years of experience in data engineering or cloud data solutions.
• Proficiency in Python, Scala, or SQL for building scalable data pipelines.
• Expertise in Databricks, Apache Spark, and Delta Lake.
• Experience with cloud platforms (AWS, Azure).
• Strong ETL, data lakes, and warehouse knowledge.
• Passion for automation, AI, and data workflows.

Responsibilities

• Build scalable data architectures – Design and implement high-performance pipelines.
• Optimise big data processing – Develop cost-effective data solutions using Databricks, Spark, and Delta Lake.
• Ensure data quality and governance – Implement data validation, lineage, and security.
• Drive automation and efficiency – Use Python, Scala, or SQL to automate workflows.
• Collaborate with cross-functional teams – Work with data scientists, analysts, and engineers.
• Stay ahead of the curve – Explore new data engineering trends, AI advancements, and cloud technologies.

Skills

Python

Scala

SQL

Databricks

Apache Spark

Delta Lake

Tools

Databricks

Apache Spark

Delta Lake

Job description

Datasmiths in Australia is seeking a Senior Data Engineer to design and optimize scalable data architectures and big data pipelines. You will work with cloud-native solutions, Databricks, Spark, and Delta Lake to deliver AI-powered insights.

The role emphasizes automation, governance, and collaboration with data scientists, analysts, and engineers to drive data-driven outcomes. Hybrid and remote options are available.