Senior Web Data Engineer for AI Pre-Training Pipelines

Cohere; Greater London; On-site; GBP 80,000 - 110,000; Full time

Benefits offered by this job

Open and inclusive culture

Weekly lunch stipend

Full health and dental benefits

401K, Pension Scheme

Parental leave top-up

Annual enrichment benefits

Remote-flexible offices

6 weeks paid vacation

Home office stipend

Job summary

Cohere is seeking a Senior Member of Technical Staff specializing in web data. The role involves developing large-scale web data pipelines that support advanced language models. You will transform raw internet data into high-quality training data and collaborate with teams to enhance processing methods.

If you're passionate about data's role in AI, this position is an opportunity to make a significant impact in a dynamic, inclusive environment.

Qualifications

• Strong software engineering skills, with proficiency in Python.
• Experience building data pipelines with large-scale web datasets.
• Knowledge of data quality assessment techniques.

Responsibilities

• Maintain large-scale pipelines for processing web corpora.
• Develop and maintain deduplication pipelines.
• Collaborate with cross-functional teams to meet data pipeline demands.

Skills

Software engineering skills

Proficiency in Python

Experience building data pipelines

Familiarity with data processing frameworks

Working with large-scale web datasets

Knowledge of data quality assessment techniques

Bridging research and engineering

Tools

Apache Spark

Apache Beam

Pandas

Job description

Cohere is seeking a Senior Member of Technical Staff specializing in web data. The role involves developing large-scale web data pipelines that support advanced language models. You will transform raw internet data into high-quality training data and collaborate with teams to enhance processing methods.

If you're passionate about data's role in AI, this position is an opportunity to make a significant impact in a dynamic, inclusive environment.