Data Engineer — Training Data

Best AI Tools Wiki; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

Unlimited PTO

Free meals

401(k) match

Job summary

OpenAI is hiring a Data Engineer to build and maintain the data pipelines that power our model training. You will develop systems for data collection, cleaning, deduplication, and quality assessment at massive scale.

Requirements include 5+ years of data engineering experience, strong Python and SQL skills, and experience with distributed data processing and frameworks like Spark or Beam. Nice to have training data experience for language models and web crawling knowledge.

Qualifications

• 5+ years of data engineering experience.
• Strong skills in Python, SQL, and distributed data processing.
• Experience with Apache Spark, Beam, or similar frameworks.
• Knowledge of data quality and governance.
• Experience with petabyte-scale data systems.

Responsibilities

• Build and maintain data pipelines powering model training.
• Develop systems for data collection, cleaning, deduplication, and quality assessment at massive scale.

Skills

Python

SQL

Distributed data processing

Apache Spark / Beam

Data quality and governance

Petabyte-scale data systems

Job description

OpenAI is hiring a Data Engineer to build and maintain the data pipelines that power our model training. You will develop systems for data collection, cleaning, deduplication, and quality assessment at massive scale.

Requirements

• 5+ years of data engineering experience
• Strong skills in Python, SQL, and distributed data processing
• Experience with Apache Spark, Beam, or similar frameworks
• Knowledge of data quality and governance
• Experience with petabyte-scale data systems

Nice to Have

• Experience with training data for language models
• Background in web crawling and text extraction
• Familiarity with data deduplication techniques
• Experience with data labeling and annotation platforms

Unlimited PTO

Free meals

401(k) match