Senior Data Infrastructure Engineer - Scalable AI Context

United States Digital Space LLC; New York, Washington (NY, District of Columbia); On-site; USD 186,000 - 233,000; Full time

Benefits offered by this job

Equity

Health insurance

Dental coverage

Vision coverage

Retirement benefits

Learning stipend

PTO

Commuter stipend

Job summary

Scale AI is hiring a Senior Software Engineer to architect and build the data infrastructure that powers our project ecosystem. You will design novel data models and processing pipelines to handle massive simulation outputs and dynamic user inputs, optimized for LLM consumption.

You will lead the data infrastructure, ensuring low latency batch processing and robust system performance while navigating ambiguous requirements and guiding junior engineers.

Qualifications

• 5+ years of backend or data infrastructure experience at senior levels.
• Expertise in systems languages and memory management for distributed architectures.
• Proven ability to design high-throughput, low-latency data pipelines.

Responsibilities

• Architect the Data Ensemble to ingest and query diverse simulation data and user inputs.
• Build scalable batch infrastructure for massive data processing with low latency.
• Model complex, relational data for ML/LLM consumption.
• Solve problems from first principles where off-the-shelf tools fail.
• Provide technical leadership and establish coding and architectural standards.

Skills

Backend infra

Senior/Staff level

Systems languages

Distributed systems

High-throughput design

Context surfacing

Tools

Rust

Go

C++

Python

Spark

PySpark

Job description

Scale AI is hiring a Senior Software Engineer to architect and build the data infrastructure that powers our project ecosystem. You will design novel data models and processing pipelines to handle massive simulation outputs and dynamic user inputs, optimized for LLM consumption.

You will lead the data infrastructure, ensuring low latency batch processing and robust system performance while navigating ambiguous requirements and guiding junior engineers.