Senior Data Ingestion Platform Engineer

United States Digital Space LLC; San Francisco (CA); On-site; USD 166,000 - 225,000; Full time

Job summary

United States Digital Space LLC is seeking an engineer for the Ingestion Core Team to build distributed platform systems that ingest high-volume, petabyte-scale data from diverse sources including cloud storage, databases, and file sources, landing into Delta Lake with CDC capabilities.

The role emphasizes reducing latency, improving throughput, and lowering costs, while designing streaming workloads and leveraging ML techniques.

Qualifications

• 5+ years of experience writing production code in one of: Java, Scala, Go, C++, or Python.
• Experience architecting, developing, and deploying large-scale distributed and asynchronous systems.
• Experience with distributed systems, streaming, Spark, databases, data processing, or CDC.
• Experience building or operating systems where scale, throughput, latency, reliability, and cost are important considerations.
• Comfortable using AI tools and defining effective evals to shorten the development loop.

Responsibilities

• Build distributed infrastructure to ingest data from diverse sources and support streaming ingestion, incremental processing, and replication. This isn't just about building plugin connectors.
• Reduce end-to-end latency, increase throughput, and reduce costs from the time data appears in source systems to when it is available in Delta Lake.
• Design and optimize streaming and distributed workloads for throughput, cost, latency, reliability, and scale.
• Optimize streaming workloads by exploring and applying ML techniques.
• Build monitoring and observability capabilities (customer-facing and internal) that provide visibility into ingestion workflows and the systems running them.
• Collaborate with partner teams to enable use cases like RAG and AI agents.
• Build Agentic SDKs with strong evaluations.

Skills

Production code

Distributed systems

AI evals

Cloud streaming

Tools

Spark

Delta Lake

Job description

United States Digital Space LLC is seeking an engineer for the Ingestion Core Team to build distributed platform systems that ingest high-volume, petabyte-scale data from diverse sources including cloud storage, databases, and file sources, landing into Delta Lake with CDC capabilities.

The role emphasizes reducing latency, improving throughput, and lowering costs, while designing streaming workloads and leveraging ML techniques.