Data Infrastructure Engineer: Scale Pipelines & Graphs

Collinear AI, Inc.; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Collinear AI, Inc. in San Francisco is seeking a Member of Technical Staff - Engineering (Data Infrastructure) to own the systems that turn large, real-world datasets into data Collinear can build on. Our environments span terabytes across archives, spreadsheets, email, PDFs, and scanned documents.

This hands-on role sits at the intersection of algorithms, distributed systems, and data quality. You will define scalable solutions for linking related records, fast search, and data cleansing, while

Qualifications

• 5+ years of experience building data-intensive systems in production.
• Experience with parallel processing of large datasets using Spark, Ray or Dask.
• Strong command of graph algorithms, and transforming large amounts of data.
• Experience with efficient string matching and pattern search at scale.
• Ability to trade off accuracy, speed, and cost, and explain decisions clearly.

Responsibilities

• Build pipelines that process multi-terabyte datasets in parallel across archives, spreadsheets, email, PDFs, and scanned documents.
• Design graph-based systems that link related records across millions of files.
• Build fast string and pattern search over large, heterogeneous datasets.
• Profile bottlenecks and decide how to split work that doesn't parallelize neatly.
• Transform data for downstream use, including replacing sensitive fields across files.
• Define data quality metrics and build review tools to catch and fix errors without full reprocessing.

Skills

Data-intensive systems

Distributed systems

Graph algorithms

String matching

Data quality

Data pipelines

Parallel processing

Systems programming (Rust/Go/C++)

Tools

Apache Spark

Ray

Dask

Job description

Collinear AI, Inc. in San Francisco is seeking a Member of Technical Staff - Engineering (Data Infrastructure) to own the systems that turn large, real-world datasets into data Collinear can build on. Our environments span terabytes across archives, spreadsheets, email, PDFs, and scanned documents.

This hands-on role sits at the intersection of algorithms, distributed systems, and data quality. You will define scalable solutions for linking related records, fast search, and data cleansing, while