Senior Data Engineer

Mytra; City of Brisbane; On-site; AUD 150,000 - 220,000; Full time

Job summary

Mytra is seeking a Senior Data Software Engineer to build the data platform used by all software and robotics systems. You will design data capture lanes, storage tiers, and pipelines across edge and cloud environments, ensuring durability, policy-driven retention, and scalable ingestion.

You will own the transformation engine, establish ID keying and crosswalks, and shape schemas and data contracts while aligning with cost and egress constraints.

Qualifications

• 5+ years building and running production backend or data infrastructure.
• Experience with columnar stores, relational databases, and No-SQL databases in production and understanding tradeoffs.
• Strong communication and ability to convey complex ideas to a diverse audience.
• Experience with Go, Python, Rust, Java or C++ in production.
• Experience with data services ingestion/CDC/event pipelines.
• Experience with ClickHouse, Parquet, Iceberg.
• Shipped services with CI/CD, monitoring, alerting.
• BS/MS in Computer Science or equivalent from having done the work.
• Ownership, accountability, and taking on responsibility.

Responsibilities

• Design the data capture lanes, storage tiers, and pipeline layout.
• Redesign the ingestion service to scale.
• Build the transform machinery and data engine you own; developers write the transforms.
• Create durable paths for what the system does and must remember.
• Build the ID keying and crosswalk tables that connect facts to site, robot, and more.
• Own retention, cost and egress policies; manage per-record retention and site-to-cloud replication.
• Run services in cloud and on customer sites with CI/CD, monitoring and alerting.

Skills

Data infrastructure

Go

Python

Rust

Java

C++

CI/CD

Monitoring

Alerting

Education

BS/MS in Computer Science

Tools

ClickHouse

Parquet

Iceberg

Kafka

NATS/JetStream

Pulsar

Job description

• Mytra is looking for a Senior Data Software Engineer to build the data platform that the rest of our software reads from
• Every robot and station puts out about 2 MB/s of telemetry and uploads a 60 MB recording every couple of minutes, roughly 6 to 8 GB per edge component per hour. We’re designing for 10,000+ storage cells a site, across many sites. Our control plane records every job, fault, and operator action. We require a system that unifies, aggregates, and transforms this data; as well as unlocking the power of our data for downstream insights
• This includes the services that pull data off the wire, the stores it lands in, the ID keying that lets you join across sources, and the pipeline machinery that turns raw capture into the tables products and agents read
• Your services will run inside customer facilities, cloud facilities, be durable for multi-day network outages, and enforces each customer’s rules for data egress. As much of the services are not built, you will also have the opportunity to architect your solution from the group up
• Design the data capture lanes, storage tiers, and pipeline layout
• Re-design the ingestion service with something that scales
• Build the transform machinery. You own the engine and what it guarantees; the team writes the transforms that run on it
• Build the durable path for what the system does and has to remember
• Build the ID keying and crosswalk tables that connect a stored fact to its site, robot, hardware revision, calibration, layout, and pallet
• Own retention, cost, and egress as policy instead of plumbing. Retention policies per record and the site-to-cloud replication path
• Run what you build. These services run in our cloud and inside customer sites, so you own their CI/CD, monitoring, and alerting

Uncertainty isn't scary, and you're willing to realize mistakes and pivotYou've used a columnar store, relational databases, and No-SQL databases in production and you have a strong understanding of how trade offs work in these technologies when used in data solutionsOpenness, inquisitiveness, and constant ambition are important valuesYou can convey complex ideas and technical concepts to a diverse audienceYou enjoy collaborative working and taking time to help othersYou've written production code in Go, Python, Rust, Java or C++You've built and run data services yourself; whether ingestion, CDC, or event pipelines, rather than writing jobs on top of someone else's platformYou know how columnar stores like ClickHouse and object-storage lakes like Parquet or Iceberg work, and have opinions about which belongs where. Schema governance, failure modes, and cost are design inputs for you, not afterthoughtsYou've shipped services and kept them running: CI/CD, monitoring, alerting, and being on the hook when they breakYou've designed schemas or data contracts that had to survive changeBS/MS in Computer Science, or the equivalent from having done the workYou seek and value ownership, accountability, and taking on responsibility5+ years building and running production backend or data infrastructureYou've worked with a durable message or streaming system (NATS/JetStream, Kafka, Pulsar, or similar) and know how delivery guarantees, backpressure, and replay actually behaveIf this role excites you, we encourage you to apply - even if you don't check every boxGCP (GCS, BigQuery/BigLake), Kubernetes, and data systems run on-prem, at the edge, or air-gapped, where you can't lean on a cloud service being reachableExperience with robotics, IoT, or industrial telemetry at volume, including formats like MCAPPrometheus and Grafana experience as a consumer of the data pipelinePrep the data store for ML or RL training to read directly