Sr. Software Engineer - Data Ingestion (Fully Remote)

Horizon3; United States; Remote; USD 120.000 - 170.000; Vollzeit

Zusammenfassung

Horizon3 seeks a data pipeline engineer to design, implement, and operate high-throughput data ingestion pipelines that transform raw pentest output into structured, queryable data at scale. You will own end-to-end delivery of pipeline features in Kubernetes and AWS.

You will partner with downstream teams to define data contracts and resolve production issues in latency, data correctness, and throughput, ensuring robust monitoring and scalable, containerized deployments across cloud environments.

Qualifikationen

• Must understand data pipeline architecture and best practices.
• Experience with scaling pipelines in cloud environments (Kubernetes/AWS).
• Familiarity with data contracts, schema design, and observability.

Aufgaben

• Design, implement, and operate high-throughput data ingestion pipelines that transform raw pentest output into structured, queryable data at scale.
• Own end-to-end delivery of pipeline features including schema design, implementation, testing, deployment, and monitoring in containerized environments like Kubernetes and AWS.
• Partner with downstream teams to define data contracts and resolve production issues in pipeline latency, data correctness, and throughput.

Kenntnisse

Data pipeline design

Distributed systems

Containerization

Monitoring and observability

Performance optimization

Tools

Kubernetes

AWS

SQL

ETL

Jobbeschreibung

• Design, implement, and operate high-throughput data ingestion pipelines that transform raw pentest output into structured, queryable data at scale.
• Own end-to-end delivery of pipeline features including schema design, implementation, testing, deployment, and monitoring in containerized environments like Kubernetes and AWS.
• Partner with downstream teams to define data contracts and resolve production issues in pipeline latency, data correctness, and throughput.