Data Engineer Aws - Real-Time Pipelines (Hybrid)
Asx
A complete application in a minute
Tailored resume and cover letter, ready to send.
Benefits offered by this job
Job summary
Endava is seeking a Lead Data Engineer to design, build and operate reliable data platforms that transform complex data into actionable business insights. You’ll work across the data lifecycle with Python/SQL and hands-on experience in Databricks, Snowflake or Microsoft Fabric.
You’ll translate business requirements into secure, scalable data solutions, guide engineering decisions and strengthen engineering practices while collaborating with clients, analysts and data scientists.
Qualifications
- Hands-on experience delivering production data solutions using one or more of Databricks, Snowflake or Microsoft Fabric.
- Strong programming skills in Python and SQL.
- Experience designing, developing and operating data pipelines, including ETL/ELT processes, orchestration and monitoring.
- Sound understanding of data architecture and modelling techniques, including analytical and dimensional modelling.
- Understanding of ontologies and semantic modelling, and how they connect data structures to business concepts.
- Experience optimizing data processing and troubleshooting pipeline, query and platform performance.
- Experience integrating data through APIs, including authentication, error handling and reliable data exchange.
- Experience applying role-based access control (RBAC) and data security best practices.
- Experience using Git, maintaining repositories, writing automated tests and building CI/CD workflows for data solutions.
- Experience developing data visualisations and reports.
- Ability to explain technical choices clearly and work with stakeholders to guide engineers.
Responsibilities
- Translate business requirements into technical solutions; provide guidance and design decisions.
- Design, build and maintain data pipelines ingesting data from APIs.
- Develop solutions using Databricks, Snowflake or Microsoft Fabric.
- Contribute to data architecture and solution design.
- Develop data models, ontologies and semantic layers for consistent business definitions.
- Implement pipeline orchestration, scheduling, monitoring and operational support.
- Troubleshoot data and pipeline issues; improve performance, scalability and cost efficiency.
- Apply data quality controls and automated testing throughout the data lifecycle.
- Implement RBAC and data security best practices.
- Apply DevOps practices to data engineering, including version control, CI/CD and automated deployment.
- Build data visualisations and reports for business users.
- Document solutions and share knowledge to support maintainable delivery.
Skills
Tools
Job description
We’re looking for a Lead Data Engineer to help our clients design, build and operate reliable data platforms that turn complex data into useful business insights.
You’ll work across the data lifecycle, from architecture and source integration through to data pipelines, semantic models, visualisation and reporting. You’ll bring strong engineering skills in Python and SQL, alongside hands‑on experience with at least one of Databricks, Snowflake or Microsoft Fabric.
Working closely with clients, analysts, data scientists and fellow engineers, you’ll translate business requirements into secure, maintainable and scalable data solutions. You’ll also provide technical guidance, contribute to design decisions and help the team strengthen its engineering practices.
- Work directly with clients to translate business requirements into technical solutions; providing guidance and contributing to design decisions.
- Design, build and maintain data pipelines to ingest, transform and deliver data from a range of sources, including APIs.
- Develop solutions using Databricks, Snowflake or Microsoft Fabric, selecting approaches that suit the client’s requirements and existing environment.
- Contribute to data architecture and solution design, including data storage, integration and modelling.
- Develop data models, ontologies and semantic layers that establish consistent business definitions and support analytics and reporting.
- Implement pipeline orchestration, scheduling, monitoring and operational support to keep data services reliable.
- Troubleshoot data and pipeline issues, improving performance, scalability and cost efficiency.
- Apply data quality controls, automated testing and validation throughout the data lifecycle.
- Implement role‑based access control and follow security best practices for data platforms and integrations.
- Apply DevOps practices to data engineering, including version control, repository management, CI/CD and automated deployment.
- Build data visualisations and reports that make information clear and useful to business users.
- Document solutions, review code and share knowledge to support maintainable delivery and continuous improvement.
What you’ll bring
- Hands‑on experience delivering production data solutions using one or more of Databricks, Snowflake or Microsoft Fabric.
- Strong programming skills in Python and SQL.
- Experience designing, developing and operating data pipelines, including ETL/ELT processes, orchestration and monitoring.
- A sound understanding of data architecture and modelling techniques, including analytical and dimensional modelling.
- An understanding of ontologies and semantic modelling, and how they connect data structures to business concepts.
- Practical experience optimising data processing and troubleshooting pipeline, query and platform performance.
- Experience integrating data through APIs, including authentication, error handling and reliable data exchange.
- Experience applying role‑based access control (RBAC) and data security best practices.
- Experience using Git, maintaining repositories, writing automated tests and building CI/CD workflows for data solutions.
- Experience developing data visualisations and reports.
- The ability to explain technical choices clearly, work collaboratively with business and technical stakeholders, and guide other engineers.
- PostgreSQL, SQL Server, Google BigQuery or Amazon Redshift.
- R or other programming languages.
- Relevant cloud services across Microsoft Azure, AWS or Google Cloud.
- Relevant certifications in Databricks, Snowflake or Microsoft Fabric.
- Flexible, hybrid working – 20 “work from anywhere” days per year on top of our hybrid policy.
- Global career pathways and non‑linear progression; move between domains, disciplines or geographies.
- Continuous learning via certifications, internal academies, tech communities and funded conferences.
- Well‑being & lifestyle allowance, loyalty leave, inclusive parental leave and employee assistance programmes.
- A vibrant, inclusive culture where diverse perspectives power innovation.
Ready to drive AI‑powered transformation? Apply today with your CV and a short cover note outlining a data‑led solution you are proud to have shaped. We look forward to exploring how your expertise can help our clients turn bold ideas into real‑world impact.
Endava is an equal‑opportunity employer. We value the differences that make us stronger and commit to transparent, merit‑based hiring.
- Monthly Lifestyle Allowance: Contribution towards health and wellbeing activities like gym memberships.
- Novated Leasing: Pre‑tax car leasing benefit for new and used cars.
- Loyalty Leave: Receive an additional day of leave on your 3rd, 4th, and 5th work anniversaries, accumulating up to a maximum of 3 extra days of leave per year.
- Inclusive Parental Leave Policy: 12 weeks of primary carer’s leave and 4 weeks of secondary carer’s leave.
- Work From Anywhere: In addition to our hybrid working policy, we also offer 20 days of working from anywhere per year.
At Endava, we’re committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best. We welcome applications from people of all backgrounds, experiences, and perspectives—because we know that inclusive teams help us deliver smarter, more innovative solutions for our customers.
Responsibilities
• Building and maintaining dbt transformation models across multiple pipeline layers — from raw ingestion through to governed, consumer‑ready datasets
• Writing Snowflake SQL for complex transformations: type casting, deduplication, incremental load logic, surrogate key generation
• Developing and debugging Apache Airflow DAGs on AWS ECS
• Contributing to AWS Lambda‑based ingestion (Python) — file validation, control file generation, S3 event handling
• Maintaining source‑to‑target mapping documentation and aligning models to business requirements and data specs
• Supporting onboarding of downstream consumers via data sharing mechanisms and event‑driven notifications
• Participating in code reviews, writing YAML‑based data quality tests, and owning tickets from design through to production
- Strong SQL
- Proficiency with dbt Core (Must have)
- Apache Iceberg Table format (Nice to have)
- Apache Airflow
- AWS – Working Knowledge of S3, Lambda, MWAA, ECS, Glue Catalogs, Lake Formation, Lambda, SQS, EventBridge
- Experience with fixed‑format or structured financial data files
- CICD with Github Actions
When you apply, you voluntarily consent to the disclosure, collection and use of your personal data for employment/recruitment and related purposes in accordance with the Tech Aalto Privacy Policy published at TechAalto’s website: Confidentiality is assured, and only shortlisted candidates will be notified for interviews.
At NCS Australia, we believe in doing technology services better. Our commitment to quality, focus on people, and willingness to challenge traditional thinking set us apart. Our team brings this belief to life by partnering with our clients and communities to make tomorrow together.
We are committed to creating an environment that prioritises innovation, collaboration, and purposeful work. Our diverse team is empowered to make a meaningful impact with curiosity, creativity and resilience to shape better outcomes. Join us and accept the challenge of creating a better tomorrow.
Work rights and background checks: To be eligible for a position with us, applicants will need to have valid work rights for Australia and be willing to undergo a comprehensive background checking process, including probity and police checks.
ASX: Powering Australia's financial markets – Why join the ASX?
When you join ASX, you’re joining a company with a strong purpose – to power a stronger economic future by enabling a fair and dynamic marketplace for all.
In your new role, you’ll be part of a leading global securities exchange with a strong brand. We are known for being a trusted market operator and an exciting data hub.
We run critical market infrastructure, with 1 in 3 people employed within technology. Yet we have a unique complexity of roles across a range of disciplines such as operations, program delivery, financial products, investor engagement, risk and compliance.
We’re proud to foster a workplace where diversity is celebrated and inclusion is part of our everyday culture. Our employee‑led networks champion LGBTIQ+ inclusion, promote gender equality, accessibility and wellbeing, inspire giving and volunteering, and celebrate cultural and religious events, creating a sense of belonging for all. As an AWEI Bronze employer and member of the Champions of Change Coalition for gender equality, we’re committed to a fair and inclusive workplace where everyone can thrive.
Your Team: The Derivatives Clearing Technology team is part of the Markets Technology division within ASX Technology. We are a trusted partner to the Clearing business, providing solutions that ensure a stable, efficient clearing and settlement facility for ASX’s derivatives trading business.
- Embedded within one of our line‑of‑business data and reporting teams, you will collaborate with stakeholders to understand data objectives and champion the use and evolution of the central data platform.
- You will work within a domain‑aligned data product team consisting of line‑of‑business data owners and SMEs working alongside engineering resources provided by Technology.
- You will be hands‑on in the ingestion of ASX’s core data assets by the product teams, their curation into source‑aligned data products and the creation of consumable products in the form of files, reports, dashboards and APIs.
- You will form part of a data engineering practice and community that will design and drive the adoption of standards, frameworks and best practices.
- You will ensure data pipelines and products are designed and built to be run within a critical market infrastructure environment by the persistent product teams who build them.
- A hands‑on engineer with deep experience working with a variety of AWS services for ingestion, curation, storage, orchestration, data management, analytics and distribution.
- Good knowledge of data modelling and how to structure data within a data warehousing environment, while following best practice data management standards.
- Solid data product architecture and solution design experience with a good working knowledge of DevSecOps practices to achieve high levels of automation across complex data pipelines.
- Experience in real‑time streaming using Confluence Kafka and AWS services.
- Experience with Airflow, AWS Glue, Apache Iceberg, Athena, RedShift and DataZone.
- Experience with Data visualization tools such as Tableau/Power BI/QuickSight.
- Solid experience in Python, Spark and Terraform/Terragrunt.
- Experience working within a platform/product operating model where frameworks and development kits are leveraged to achieve standardisation, re‑use and scale.
- A working knowledge of data mesh/data product concepts with federated data ownership at the domain‑level but operating on centralised platforms and governance.
- Experience working in Capital Markets domain and understanding of derivative products.
We will arrange for successful candidates to have background checks, including reference and police checks, completed as part of the on‑boarding process.
Design and build Data Model Objects (DMOs), ... (continued content continues with detailed responsibilities and skills).
INGRITY is seeking an experienced Data Engineer to join a major financial services organisation undergoing a significant data modernisation program.
You'll work within a modern Databricks environment, building scalable data pipelines, transforming complex datasets and supporting the migration of legacy data platforms to a cloud‑based lakehouse architecture.
- Design, develop and optimise data pipelines using Databricks, Python/PySpark and SQL.
- Build and maintain data transformations using dbt and Delta Lake.
- Develop scalable data models across Bronze, Silver and Gold layers.
- Support data migration from legacy platforms, including SAS.
- Implement data quality checks, testing and performance optimisation.
- Collaborate with data engineers, analysts and business stakeholders.
- Strong hands‑on experience with Databricks, PySpark and SQL.
- Experience with dbt, Delta Lake and lake house architecture.
- Proven experience developing and maintaining ETL/ELT pipelines.
- Experience with cloud platforms, ideally Azure or AWS.
- Understanding of CI/CD, data testing and production support.
- Experience in insurance or financial services highly regarded.
- Exposure to AI‑assisted development tools such as GitHub Copilot or Databricks Genie advantageous.
- Modern Databricks and cloud technology environment.
- Collaborative engineering team with hybrid flexibility.
- Opportunity to contribute to AI‑enabled data engineering practices.
Location: Sydney / Brisbane (Hybrid) Contract: 12 Months Industry: Financial Services / Insurance
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
