Data Acquisition Engineer: Scalable Crawling & Ingestion

OpenAI

California (MO)

On-site

USD 140,000 - 200,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

OpenAI is seeking a skilled Software Engineer to join the Data Acquisition team within the Foundations organization. You will own and lead engineering projects around data collection, web crawling, ingestion, and search to support our model training operations.

You will collaborate with Data Processing, Architecture, and Scaling teams and work closely with the legal team on compliance and privacy matters. The role requires building scalable distributed systems and deploying backend services in a

Qualifications

BS/MS/PhD in Computer Science or related field.
4+ years of industry software development experience.
Experience with large web crawlers is a plus.
Strong background in large stateful distributed systems and data processing.

Responsibilities

Own and lead engineering projects in data acquisition incl. web crawling, data ingestion, and search.
Collaborate with Data Processing, Architecture, and Scaling teams to ensure data flow.
Coordinate with legal for data privacy/compliance matters.
Develop and deploy scalable distributed systems handling petabytes of data.
Architect and implement data indexing and search capabilities.
Build and maintain backend data storage services (key-value stores).
Deploy solutions in Kubernetes IaC environment and run routine checks.
Conduct experiments on data to gain performance insights.