Data Ingestion Engineer for Open-Scale AI Pipelines

B Capital; San Francisco (CA); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Top-tier compensation

Comprehensive medical, dental, and vision insurance

Fully paid parental leave

Paid time off

Daily lunch and dinner provided

Job summary

B Capital is seeking a Data Engineer for their San Francisco office to build and operate data ingestion systems. The role requires collaboration with researchers to enhance model performance through effective data management.

The ideal candidate has experience in web crawling technologies and enjoys experimenting with data acquisition systems. The position offers competitive compensation and an array of benefits focused on health, wellness, and work-life balance.

Qualifications

• Experience building web crawling, data ingestion, or large-scale data acquisition systems.
• Familiarity with training and evaluating LLMs.
• Comfortable working with large datasets and improving systems.

Responsibilities

• Build and operate large-scale data ingestion systems.
• Run experiments on crawling strategies and ingestion methods.
• Analyze ingested data for quality improvements.

Skills

Web crawling

Data ingestion

Large-scale data acquisition systems

Ray, Beam, Spark

Experimental design

Communication skills

Job description

B Capital is seeking a Data Engineer for their San Francisco office to build and operate data ingestion systems. The role requires collaboration with researchers to enhance model performance through effective data management.

The ideal candidate has experience in web crawling technologies and enjoys experimenting with data acquisition systems. The position offers competitive compensation and an array of benefits focused on health, wellness, and work-life balance.