AI-Driven Web Data Crawling Engineer

Rubick.AI

Oregon

Hybrid

USD 90,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Rubick is seeking a data acquisition engineer to build and maintain its data extraction engine for product information across eCommerce sites and marketplaces. You will develop scalable crawlers, validate datasets, and collaborate with cross-functional teams to ensure timely data collection.

You’ll leverage AI-driven extraction techniques and modern frameworks to improve accuracy, speed, and reliability, while contributing to reusable pipelines and knowledge bases across the data acquisition

Qualifications

  • 1+ year of experience in Web Crawling, Data Scraping, Product Matching, Data Extraction, or similar roles.
  • Strong proficiency in Python for web scraping and automation.
  • Hands-on experience with BeautifulSoup, Scrapy, Selenium, Playwright, or similar frameworks.
  • Good understanding of HTML, CSS, XPath, JSON, and DOM structures.
  • Strong analytical and problem-solving skills.
  • Ability to work with large datasets while maintaining high data quality.
  • Experience in eCommerce, Retail, Marketplace Operations, or Catalog Management is preferred.
  • Exposure to AI-assisted data extraction, Product Intelligence, or Machine Learning data pipelines is a strong advantage.

Responsibilities

  • Develop, maintain, and optimize web crawlers to extract product data from eCommerce websites and marketplaces.
  • Collect structured and unstructured product information, including product details, pricing, images, specifications, and availability.
  • Clean, validate, and organize extracted datasets to ensure accuracy and consistency.
  • Monitor crawler performance, identify failures, and resolve data extraction issues.
  • Collaborate with Product, Engineering, and Data teams to ensure reliable and timely data collection.
  • Maintain documentation for crawling processes, extraction rules, and data quality standards.
  • Leverage AI-powered extraction techniques to improve data accuracy and extraction efficiency.
  • Build intelligent crawling workflows using Python and modern web automation frameworks.
  • Implement automated data validation and monitoring systems to maintain high-quality datasets.
  • Collaborate with AI and Data Engineering teams to support Machine Learning models and Product Intelligence systems.
  • Continuously optimize crawling performance for speed, scalability, and reliability.
  • Identify opportunities to automate repetitive extraction and validation workflows.
  • Improve data collection strategies by adopting new crawling technologies and AI-assisted solutions.
  • Build reusable crawling frameworks and standardized extraction pipelines.
  • Contribute to knowledge repositories, documentation, and process improvements across the data acquisition function.

Skills

Python for web scraping
Data processing
Web crawling
Problem-solving

Tools

BeautifulSoup
Scrapy
Selenium
Playwright

Job description

Rubick is seeking a data acquisition engineer to build and maintain its data extraction engine for product information across eCommerce sites and marketplaces. You will develop scalable crawlers, validate datasets, and collaborate with cross-functional teams to ensure timely data collection.

You’ll leverage AI-driven extraction techniques and modern frameworks to improve accuracy, speed, and reliability, while contributing to reusable pipelines and knowledge bases across the data acquisition

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Crawling - Consultant
Data Crawling - Consultant

Rubick • Oregon

Hybrid
USD 90,000 - 150,000
Web Scraping Engineer - Scale & AI-Powered Crawlers
Web Scraping Engineer - Scale & AI-Powered Crawlers

Source.ai • United States

Remote
USD 60,000 - 94,000
Health insurance
Psychotherapy
Wellness app
+6
Remote Research Engineer - Web Crawling & Data Pipelines
Remote Research Engineer - Web Crawling & Data Pipelines

ElevenLabs Inc. • Maine

Hybrid
USD 140,000 - 210,000
Discretionary stipend
Travel stipend
Co-working stipend
+1
Remote Data Infrastructure Engineer - Web Crawling Pipelines
Remote Data Infrastructure Engineer - Web Crawling Pipelines

Unknown • United States

Remote
USD 160,000 - 250,000
Fully remote role
Remote Research Web Crawler Engineer
Remote Research Web Crawler Engineer

Wynd Labs • New York (NY)

Hybrid
USD 120,000 - 180,000
Research Crawling Engineer
Research Crawling Engineer

Startup Talents • London (KY)

Remote
USD 150,000 - 225,000
Research Crawling Engineer
Research Crawling Engineer

Wynd Labs • New York (NY)

Hybrid
USD 120,000 - 180,000
AI-Driven Sales Tech Product Leader
AI-Driven Sales Tech Product Leader

Rubrik • Palo Alto (CA)

Hybrid
USD 193,000 - 290,000
Remote Web Scraper for AI Training Data
Remote Web Scraper for AI Training Data

ESR Healthcare • United States

Remote
USD 28,000 - 69,000
High-Impact Web Scraping Engineer
High-Impact Web Scraping Engineer

String • United States

Remote
USD 60,000 - 90,000