Data Crawling - Consultant

Rubick

Oregon

Hybrid

USD 90,000 - 150,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Rubick is seeking a data acquisition engineer to build and maintain its data extraction engine for product information across eCommerce sites and marketplaces. You will develop scalable crawlers, validate datasets, and collaborate with cross-functional teams to ensure timely data collection.

You’ll leverage AI-driven extraction techniques and modern frameworks to improve accuracy, speed, and reliability, while contributing to reusable pipelines and knowledge bases across the data acquisition

Qualifications

  • 1+ year of experience in Web Crawling, Data Scraping, Product Matching, Data Extraction, or similar roles.
  • Strong proficiency in Python for web scraping and automation.
  • Hands-on experience with BeautifulSoup, Scrapy, Selenium, Playwright, or similar frameworks.
  • Good understanding of HTML, CSS, XPath, JSON, and DOM structures.
  • Strong analytical and problem-solving skills.
  • Ability to work with large datasets while maintaining high data quality.
  • Experience in eCommerce, Retail, Marketplace Operations, or Catalog Management is preferred.
  • Exposure to AI-assisted data extraction, Product Intelligence, or Machine Learning data pipelines is a strong advantage.

Responsibilities

  • Develop, maintain, and optimize web crawlers to extract product data from eCommerce websites and marketplaces.
  • Collect structured and unstructured product information, including product details, pricing, images, specifications, and availability.
  • Clean, validate, and organize extracted datasets to ensure accuracy and consistency.
  • Monitor crawler performance, identify failures, and resolve data extraction issues.
  • Collaborate with Product, Engineering, and Data teams to ensure reliable and timely data collection.
  • Maintain documentation for crawling processes, extraction rules, and data quality standards.
  • Leverage AI-powered extraction techniques to improve data accuracy and extraction efficiency.
  • Build intelligent crawling workflows using Python and modern web automation frameworks.
  • Implement automated data validation and monitoring systems to maintain high-quality datasets.
  • Collaborate with AI and Data Engineering teams to support Machine Learning models and Product Intelligence systems.
  • Continuously optimize crawling performance for speed, scalability, and reliability.
  • Identify opportunities to automate repetitive extraction and validation workflows.
  • Improve data collection strategies by adopting new crawling technologies and AI-assisted solutions.
  • Build reusable crawling frameworks and standardized extraction pipelines.
  • Contribute to knowledge repositories, documentation, and process improvements across the data acquisition function.

Skills

Python for web scraping
Data processing
Web crawling
Problem-solving

Tools

BeautifulSoup
Scrapy
Selenium
Playwright

Job description

You will be responsible for building and maintaining Rubick's data acquisition engine by extracting, validating, and structuring product information from eCommerce websites and marketplaces. This role focuses on developing scalable web crawling solutions that power Product Discovery, Search, Catalog Intelligence, and Market Intelligence platforms.

The role involves three areas: Part 1: The Fundamentals | Part 2: AI-Driven Data Crawling Excellence | Part 3: Innovation & Improvements

Part 1: The Fundamentals
  • Develop, maintain, and optimize web crawlers to extract product data from eCommerce websites and marketplaces.
  • Collect structured and unstructured product information, including product details, pricing, images, specifications, and availability.
  • Clean, validate, and organize extracted datasets to ensure accuracy and consistency.
  • Monitor crawler performance, identify failures, and resolve data extraction issues.
  • Collaborate with Product, Engineering, and Data teams to ensure reliable and timely data collection.
  • Maintain documentation for crawling processes, extraction rules, and data quality standards.
Part 2: AI-Driven Data Crawling Excellence
  • Leverage AI-powered extraction techniques to improve data accuracy and extraction efficiency.
  • Build intelligent crawling workflows using Python and modern web automation frameworks.
  • Implement automated data validation and monitoring systems to maintain high-quality datasets.
  • Collaborate with AI and Data Engineering teams to support Machine Learning models and Product Intelligence systems.
  • Continuously optimize crawling performance for speed, scalability, and reliability.
  • Identify opportunities to automate repetitive extraction and validation workflows.
  • Improve data collection strategies by adopting new crawling technologies and AI-assisted solutions.
  • Build reusable crawling frameworks and standardized extraction pipelines.
  • Contribute to knowledge repositories, documentation, and process improvements across the data acquisition function.
To Have
  • 1+ year of experience in Web Crawling, Data Scraping, Product Matching, Data Extraction, or similar roles.
  • Strong proficiency in Python for web scraping and automation.
  • Hands‑on experience with BeautifulSoup, Scrapy, Selenium, Playwright, or similar frameworks.
  • Good understanding of HTML, CSS, XPath, JSON, and DOM structures.
  • Strong analytical and problem‑solving skills.
  • Ability to work with large datasets while maintaining high data quality.
  • Experience in eCommerce, Retail, Marketplace Operations, or Catalog Management is preferred.
  • Exposure to AI-assisted data extraction, Product Intelligence, or Machine Learning data pipelines is a strong advantage.
About Rubick : Company Overview

Rubick OS is a leading platform for brands and marketplaces, enabling seamless management of catalog automation, pricing and competitive intelligence, product listing intelligence, CAST systems, and brand and assortment intelligence.

Our solutions are used across five countries by major brands and marketplaces, including Amazon, Myntra, Reliance Group, Nykaa, Flipkart, Rare Rabbit, Decathlon, Celio, The Bay, Kiabi, Myer, Jumbo, and many more.

Work Model

Work from office – Monday to Saturday

Team Size

250+ Employees

Investors

Invested by Innospark US, Betatron Hong Kong, MJV India, 1Crowd India , and other investors.

Rubick.ai is a place to work if you want to build something foundational rather than incremental—it’s aiming to become the AI operating layer for ecommerce, solving complex, real-world problems across cataloging, pricing, and marketplace operations at scale.

You get high ownership, direct impact, and exposure to various parts of the business, making you an entrepreneur in-house. Rubick operates in the AI segment, which means you will be at the forefront of the technological revolution.

Rubick is profitable and growing and is looking to work with 10,000 brands in the next 2–3 years .

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr. Sales Engineer - AI
Sr. Sales Engineer - AI

Rubrik • Colorado

On-site
USD 150,000 - 240,000
Staff Product Manager, IT - GTM
Staff Product Manager, IT - GTM

Rubrik • Palo Alto (CA)

Hybrid
USD 193,000 - 290,000
Senior Director, IT Product Management — GTM & Customer Experience
Senior Director, IT Product Management — GTM & Customer Experience

Rubrik • Palo Alto (CA)

On-site
USD 278,000 - 390,000
Product Growth Intern (MBA), Summer 2027
Product Growth Intern (MBA), Summer 2027

Rubrik • Palo Alto (CA)

On-site
USD 120,000 - 160,000
Senior Manager, Quoting Strategy & Operations
Senior Manager, Quoting Strategy & Operations

Rubrik • Palo Alto (CA)

On-site
USD 173,000 - 260,000
AMER Deal Structuring Manager
AMER Deal Structuring Manager

Rubrik • Frankfort (KY)

Remote
USD 126,000 - 210,000
Head of Sales Strategy & Operations
Head of Sales Strategy & Operations

Rubrik • United States

Remote
USD 152,000 - 180,000
AMER Deal Structuring Manager
AMER Deal Structuring Manager

Rubrik • Hartford (CT)

Remote
USD 126,000 - 189,000
AMER Deal Structuring Manager
AMER Deal Structuring Manager

Rubrik • Nashville (TN)

Remote
USD 126,000 - 210,000
Bonus potential
Equity
Benefits
Senior AI Engineer, Agentic Systems
Senior AI Engineer, Agentic Systems

Rubrik • Palo Alto (CA)

On-site
USD 189,000 - 283,000