Senior Web Data Engineer for AI Training Pipelines

Cohere; New York (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Lunch stipend

Health and dental benefits

RRSP matching / 401K

Parental leave top-up

Enrichment benefits (arts & culture, ɫ

6 weeks of paid vacation

Travel budget + annual offsite

Home office stipend

Job summary

Cohere is seeking a Senior Member of Technical Staff, Web Data, in New York or remote, to own large-scale web data pipelines for pre-training language models. You will transform raw web data into high-quality training data, implement extraction, parsing, deduplication and filtering, and collaborate with data and evaluation teams.

We value Python proficiency, experience with Spark/Beam, and a track record of handling large datasets.

Qualifications

• Strong software engineering skills with Python and building data pipelines.
• Familiarity with Spark, Beam, Pandas, or similar data processing frameworks.
• Experience working with large-scale web datasets.
• Knowledge of data quality assessment techniques and data mixtures experimentation.
• Passion for bridging research and engineering to solve data challenges in AI model training.

Responsibilities

• Maintain large-scale pipelines for processing web corpora.
• Work on filtering and quality-scoring systems to identify high-value web documents.
• Analyze web data composition across domains, languages and time periods.
• Develop and maintain highly-performant deduplication pipelines.
• Collaborate with cross-functional teams to ensure data pipelines meet the demands of cutting-edge language models.

Skills

Python

Data pipelines

Tools

Apache Spark

Apache Beam

Pandas

Job description

Cohere is seeking a Senior Member of Technical Staff, Web Data, in New York or remote, to own large-scale web data pipelines for pre-training language models. You will transform raw web data into high-quality training data, implement extraction, parsing, deduplication and filtering, and collaborate with data and evaluation teams.

We value Python proficiency, experience with Spark/Beam, and a track record of handling large datasets.