Senior Data Pipeline Engineer, Web Pretraining

Cohere; Greater London; Remote; GBP 110,000 - 150,000; Full time

Benefits offered by this job

Lunch stipend

Health benefits

Parental Leave

Education stipend

Vacation time

Home office stipend

Job summary

Cohere is seeking a Senior Member of Technical Staff focused on web data for pre-training. You will design and own the data pipeline, transforming raw, noisy internet data into clean training data for language models.

You will optimize extraction, parsing, deduplication and filtering while assessing data quality to improve model performance. You will partner with data and evaluation teams to iterate on the corpus, ensuring robust, scalable pipelines that support Cohere’s mission in enterprise

Qualifications

• Proficient in Python with experience building data pipelines.
• Experience with large-scale web datasets and data quality assessment.
• Familiar with Spark, Beam, Pandas or similar data processing frameworks.

Responsibilities

• Maintain large-scale web data pipelines.
• Develop filtering and quality-scoring for web documents.
• Analyze data composition across domains, languages and time.
• Create and optimize deduplication pipelines.
• Collaborate with researchers and engineers to support model training.

Skills

Python

Data pipelines

Software engineering

Web data

Data quality

Research bridging

Large-scale data

Tools

Apache Spark

Apache Beam

Pandas

Job description

Cohere is seeking a Senior Member of Technical Staff focused on web data for pre-training. You will design and own the data pipeline, transforming raw, noisy internet data into clean training data for language models.

You will optimize extraction, parsing, deduplication and filtering while assessing data quality to improve model performance. You will partner with data and evaluation teams to iterate on the corpus, ensuring robust, scalable pipelines that support Cohere’s mission in enterprise