Data Engineer

Vericent Pty; City of Melbourne; On-site; AUD 120,000 - 160,000; Full time

Job summary

Vericent in Melbourne is seeking an experienced Data Engineer to design, build and optimise enterprise-scale data pipelines and cloud-based data solutions that enable analytics, reporting and AI-driven outcomes.

You will collaborate with data scientists, software engineers and business stakeholders to ensure data quality, security and performance across the data lifecycle, while contributing to CI/CD practices and production support.

Qualifications

• Advanced SQL skills and performance awareness for large data sets.
• Experience with data modelling, warehousing and analytical patterns.
• Experience implementing data-quality controls and reconciliation processes.
• Strong documentation and stakeholder communication.

Responsibilities

• Design, build and maintain batch and real-time data pipelines.
• Develop ingestion and transformation processes for structured and semi-structured data.
• Build datasets for reporting, analytics and ML.
• Develop solutions using SQL, Python, dbt, PySpark or equivalent technologies.
• Integrate data from enterprise applications and external sources.
• Optimize pipelines for reliability, performance and cost.
• Support migration and modernization of legacy data processes.

Skills

Advanced SQL

Data modelling

Data warehousing

Strong stakeholder communication

Technical documentation

Education

Tertiary qualification in computer science, engineering, information systems or related discipline

Tools

Snowflake

dbt

Databricks

PySpark

Git

CI/CD

Job description

Data Architecture Melbourne Australia Regular / Full time

Description

About the Role

Vericent is partnering with a leading global technology consulting organisation to recruit an experienced Data Engineer. This is an exciting opportunity to work on enterprise-scale data initiatives, building modern data pipelines and cloud-based data solutions that enable analytics, reporting, and AI-driven outcomes.

In this role, you'll collaborate with cross-functional teams, including data scientists, software engineers, product owners, and business stakeholders, to design, develop, and optimise scalable data integration and transformation solutions. You'll also play a key role in ensuring data quality, reliability, security, and performance across the entire data lifecycle.

Key Responsibilities

• Design, build and maintain batch and near-real-time data pipelines.
• Develop ingestion and transformation processes for structured and semi-structured data.
• Build reusable data components and curated datasets for reporting, analytics and machine learning.
• Develop solutions using SQL, Python, dbt, PySpark or equivalent technologies.
• Integrate data from enterprise applications, APIs, event streams and external sources.
• Optimise data pipelines for reliability, performance and cost.
• Support migration and modernisation of legacy data processes.

Data Modelling and Quality

• Develop logical and physical data models aligned to business requirements.
• Build dimensional, relational and analytical data structures.
• Implement data-quality rules, reconciliations and validation controls.
• Investigate data defects and coordinate remediation with source-system and product teams.
• Maintain data definitions, lineage, metadata and technical documentation.
• Apply appropriate handling for sensitive, customer and commercially restricted data.

Engineering and DevOps Practices

• Develop maintainable, tested and version-controlled data solutions.
• Implement automated testing across ingestion, transformation and data-quality processes.
• Support CI/CD pipelines and controlled deployment across development, test and production.
• Participate in code reviews and apply engineering standards.
• Implement monitoring, alerting and operational logging.
• Reduce manual deployment and support activities through automation.

Production Support and Operations

• Monitor production pipelines, schedules and data-processing workloads.
• Investigate and resolve failed jobs, delayed data and data-quality incidents.
• Participate in incident, problem and change-management processes.
• Perform root-cause analysis and implement permanent remediation.
• Maintain operational runbooks, support procedures and recovery steps.
• Work with Platform Operations to ensure data products meet availability and service-level requirements.

Collaboration and Delivery

• Work with product owners and stakeholders to define data requirements and acceptance criteria.
• Collaborate with data scientists to provide model-ready datasets and features.
• Work with software engineers to support application and API data requirements.
• Contribute to technical design, estimation and delivery planning.
• Communicate data risks, dependencies and design decisions clearly.

Required Skills and Experience

• Advanced SQL skills and strong understanding of data structures and query performance.
• Experience with at least two of the following:
• Snowflake
• dbt
• Databricks and PySpark
• Experience building automated data-ingestion and transformation pipelines.
• Strong understanding of data modelling, warehousing and analytical data patterns.
• Experience implementing data-quality controls and reconciliation processes.
• Experience with Git, code reviews and CI/CD practices.
• Understanding of cloud security, access control and data-protection requirements.
• Experience supporting production data pipelines and resolving operational incidents.
• Strong technical documentation and stakeholder-communication skills.
• Tertiary qualification in computer science, engineering, information systems or a related discipline, or equivalent practical experience.

Key Deliverables and Success Measures

• Reliable and scalable data pipelines and data products.
• Achievement of agreed data availability and processing service levels.
• Reduction in pipeline failures and recurring data-quality issues.
• Accurate, traceable and well-documented datasets.
• Automated testing, deployment, monitoring and recovery processes.
• Improved pipeline performance and cloud-cost efficiency.
• Compliance with security, data-governance and engineering standards.
• Positive engagement with product, platform, analytics and business stakeholders.