Lead Data Engineer

LTM; Sydney; In loco; AUD 150.000 - 210.000; Tempo pieno

Descrizione del lavoro

LTM in Sydney, Australia, seeks a senior data engineering lead to architect and deliver a Transactions Ingestion Data Platform on GCP, covering real-time and batch pipelines. You will work with the customer’s data teams to shape schemas for Spanner and BigQuery and drive MVP delivery.

Hands-on with Python/Java, CI/CD, Terraform, Cloud Composer, and data governance tools; experience with generative AI integrations in large enterprise environments is valued.

Competenze

• In-depth knowledge of GCP data and analytics technologies.
• Experience designing large-scale transaction data platform architectures on GCP.
• Define solution architecture and detailed design, and perform hands-on implementation for data pipelines.

Mansioni

• Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform (Real Time and Batch).
• Interact with customer’s data engineering team to provide guidance on platform architecture, schemas (Spanner/BigQuery) and solutions.
• Develop and unit test ingestion/processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery) and handover to customer team.
• Perform pair programming with customer’s data engineers.
• Project manage sprint-based engagements, leading planning, backlog management, scope, risks, and MVP delivery.
• Collaborate with cross-disciplinary project team as tech lead for workstreams or entire project.

Conoscenze

GCP data analytics

Data architecture

Data pipelines

Python

Java

CI/CD

Cloud Composer

Dataplex/Data Catalog

Strumenti

Terraform

Dataform

dbt

Kafka

Google Dataflow

Spanner

BigQuery

Descrizione del lavoro

• Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch
• Interact with customer's data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development
• Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer's technical team
• Perform pair programming with customer's data engineers
• Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery
• Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program

Minimum Qualifications

• In-depth knowledge and experience of GCP data and analytics technologies
• Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
• Define solution architecture and detailed design, and perform hands-on implementation for
• data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.
• real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
• Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
• Orchestration setup and Cloud Composer DAG development for data pipeline workflows
• Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog) Experience applying Generative AI technologies and integrations in enterprise environments.

Tools and languages experience Required

Must have:

• Must have: programming knowledge and willingness to be hands-on - Python, Java

Good to Have:

• Experience with dbt or Dataform, Terraform in context of data pipelines
• Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka, HBASE
• Understanding and experience with NoSQL databases such as HBASE, MongoDB
• Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations Certification
• Good to have GCP Data engineering

LTM is proud to be an equal opportunity employer. We are committed to equal employment opportunity regardless of race, ethnicity, nationality, gender, gender-identity, gender expression, language, age, sexual orientation, religion, marital status, veteran status, socio-economic status, disability or any other characteristic protected by applicable law.