Technical Lead, Data Engineering

Voreas Laboratories Inc.; Washington, Northern (District of Columbia, KY); Hybrid; USD 180,000 - 230,000; Full time

Benefits offered by this job

Fully remote

Flexible hours

Design ownership

Modern stack

Job summary

Voreas Laboratories Inc. is hiring an Engineering – Technical Lead, Data Engineering to own the schemas, the graph data model, and the detection framework for our data lake platform in Washington, DC. This full-time role includes writing detections and advancing the platform architecture.

You will work with a stack including Amazon EKS, Spark, Iceberg, Delta Lake, Argo, and Python/Scala, while enabling scalable pipelines and production-grade data pipelines. Remote within the US is supported.

Qualifications

• Production Spark against an open table format, including table versioning.
• Schema design that acts as a contract consumed by multiple teams.
• Data modeling that balances storage and query workloads.
• Experience with CI/CD and infrastructure as code.

Responsibilities

• Define canonical table structures for the platform and the evidence model.
• Own the graph data model and how detections are expressed and aged.
• Build a shared four-part detection framework for consistency.
• Define detector interfaces and ensure production feasibility.
• Develop detections within the framework and collaborate with security researchers.

Skills

Production Spark

Schema design

Data modeling

CI/CD

Tools

Amazon EKS

Argo Workflows

Argo CD

Crossplane

Python

Scala

Apache Iceberg

Delta Lake

Job description

Engineering – Technical Lead, Data Engineering – Washington, DC Preferred – Full-Time

DNS and netflow telemetry lands in a data lake in our own AWS account, detection pipelines score it, and the findings are written into an intelligence graph. Security researchers design the detections and often build them. This team builds everything around them: the pipelines that reduce lake data to what a detector needs, the packaging that lets it run on our platform, the orchestration that runs it on schedule at full volume, and the path that writes what it found back to the lake as evidence.

Our stack is Amazon EKS, Spark, Apache Iceberg and Delta Lake, Argo Workflows, Argo CD, Crossplane, and Python and Scala. We do not expect you to have used all of it. We do expect that you have worked in each of these areas, in whatever tools you used at the time: continuous integration and delivery, infrastructure as code, distributed data processing, container orchestration, a data lake on an open table format, and Python.

This role owns the schemas, the graph data model, and the detection framework the rest of the platform is built on. You will also write detections yourself. This is not a pure architecture role.

Location
Washington, DC Preferred

Committment
Full-Time

Work style
Remote

Responsibilities

• The schemas. Define the canonical table structures the whole platform reads and writes, starting with the evidence model every detection writes to and every consumer reads from. Getting it wrong is expensive in a specific way: a finding written without a record of who produced it, when, and what it was based on cannot be repaired afterwards. You also own the detection type catalog — what each detection asserts, how it is expressed, and how long it stays relevant.
• The graph data model. What becomes a node, what becomes an edge, what context lives on each, how a new detection type is added without breaking anything already written, and how evidence ages out over time. The design language is settled and documented; the open questions are real and you will be answering them.
• The detection framework. Every detection needs the same four things around it: data reduced to its input grain, a container contract it conforms to, a workflow that runs it, and a path that writes its output back as evidence. You build that once, as shared code, so detections written by several people over several years do not each solve those four problems differently. This is the difference between four pipelines and forty.
• The interface with security research. Define what a detector receives, what it must emit, and what it may assume about the data it is fed. Researchers build against that contract, engineers build the pipeline around it, and you are responsible for whether it holds at production volume.
• Detections. You will build on the framework you design, not only specify it.

Required Qualifications

• Production Spark against an open table format, including table versioning. We run both Apache Iceberg and Delta Lake.
• You have designed a schema that other teams consumed as a contract, and lived through evolving it afterward.
• You have modeled data for a query workload, not only for storage. What the access patterns are, and what the model costs you when they change.
• You have written a specification that someone outside your team implemented against, and been responsible for whether what came back was acceptable.
• You have built internal libraries or frameworks that other engineers built on top of.
• Working experience across continuous integration and delivery, infrastructure as code, distributed data processing, container orchestration, a data lake on an open table format, and Python.
• Must be a US citizen

Preferred Qualifications

• Our specific stack: Amazon EKS, Argo Workflows, Argo CD, Crossplane.
• Scala, alongside Python.
• Graph data modeling, including how entities and relationships age out of a store.

Why Join Us

• Fully remote within the United States, with flexible working hours
• Foundational design ownership over the models the whole platform depends on
• A modern stack and an engineering culture that values automation and best practices