AI-Driven Data Crawling Specialist (Remote)

Rubick; Oregon (WI); Hybrid; USD 90,000 - 130,000; Full time

Benefits offered by this job

Work from home - Monday to Saturday

Job summary

Rubick is seeking a data crawling engineer to build and maintain its data acquisition engine for extracting, validating, and structuring product information from eCommerce sites. This role powers Product Discovery, Search, Catalog Intelligence, and Market Intelligence platforms.

You will develop scalable crawlers using Python and frameworks like BeautifulSoup, Scrapy, Selenium, and Playwright, focusing on AI-assisted extraction, data validation, and high-quality data pipelines.

Qualifications

• Proficient in Python for web scraping and automation.
• Hands-on experience with HTML/CSS and DOM structures.
• Experience with data validation, cleaning, and structuring of datasets.
• Familiarity with AI-assisted data extraction and ML data pipelines is a plus.
• Able to handle large datasets while maintaining data quality.
• Experience in eCommerce, retail, or marketplace data is preferred.

Responsibilities

• Develop, maintain, and optimize web crawlers to extract product data from eCommerce sites.
• Collect structured and unstructured product information including details, pricing, images, specs, and availability.
• Clean, validate, and organize extracted datasets for accuracy and consistency.
• Monitor crawler performance and resolve extraction issues.
• Collaborate with Product, Engineering, and Data teams to ensure timely data collection.
• Maintain documentation for crawling processes, extraction rules, and data quality standards.
• Leverage AI-powered extraction techniques to improve accuracy and efficiency.
• Build intelligent crawling workflows using Python and modern web automation frameworks.
• Implement automated data validation and monitoring systems for high-quality datasets.
• Collaborate with AI/Data Engineering teams to support ML models and Product Intelligence systems.
• Continuously optimize crawling performance for speed, scalability, and reliability.
• Identify opportunities to automate repetitive extraction and validation workflows.
• Improve data collection strategies by adopting new crawling technologies and AI-assisted solutions.
• Build reusable crawling frameworks and standardized extraction pipelines.
• Contribute to knowledge repositories and process improvements across data acquisition.

Skills

Python for scraping

HTML/CSS/DOM

Data quality & validation

Analytical thinking

Large datasets

E-commerce domain knowledge

AI-assisted extraction exposure

Tools

BeautifulSoup

Scrapy

Selenium

Playwright

Job description

Rubick is seeking a data crawling engineer to build and maintain its data acquisition engine for extracting, validating, and structuring product information from eCommerce sites. This role powers Product Discovery, Search, Catalog Intelligence, and Market Intelligence platforms.

You will develop scalable crawlers using Python and frameworks like BeautifulSoup, Scrapy, Selenium, and Playwright, focusing on AI-assisted extraction, data validation, and high-quality data pipelines.