Data Engineer, Scrape and Analysis Pipeline

Questah Staffing

United States

Remote

USD 90,000 - 150,000

Full time

8 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

BestWaitlistApps.com is seeking an engineer to maintain the scraping and analysis pipeline that powers feature ratings across vendors and write results into the comparison data. The role involves keeping the scraper up to date, running analysis on scraped text and screenshots, and generating logs and gap reports for held-out features, with a focus on reliability and traceability.

You will collaborate with product and data teams, review discrepancies between AI analysis and stored feature marks,

Qualifications

  • Experience with PHP and Laravel console commands.
  • Experience with Node and a headless browser for scraping.
  • Comfortable with SQL and schema migrations.

Responsibilities

  • Maintain the scraper that renders each vendor's pages and takes screenshots.
  • Maintain the commands that analyze the scraped text and the screenshots and write the detected features to the comparison tables.
  • Compare the AI analysis with the stored feature marks and report where they disagree.
  • Maintain the gap report that lists features that tracked products have and the catalog lacks.
  • Fix failed scrapes and keep a log of what each scrape stored.

Tools

PHP
Laravel
Node.js
Headless browser
SQL
Schema migrations

Job description

Maintain the scraping and analysis commands that collect the evidence behind the BestWaitlistApps.com feature ratings and write it to the comparison data.

This role is with BestWaitlistApps.com.

Nice to have
  • Experience calling large language model APIs from application code.
  • Experience with image analysis.
Responsibilities
  • Maintain the scraper that renders each vendor's pages and takes screenshots.
  • Maintain the commands that analyze the scraped text and the screenshots and write the detected features to the comparison tables.
  • Compare the AI analysis with the stored feature marks and report where they disagree.
  • Maintain the gap report that lists features that tracked products have and the catalog lacks.
  • Fix failed scrapes and keep a log of what each scrape stored.
Requirements
  • Experience with PHP and Laravel console commands.
  • Experience with Node and a headless browser for scraping.
  • Comfortable with SQL and schema migrations.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer: Web Scraping & AI Feature Analysis
Data Engineer: Web Scraping & AI Feature Analysis

Questah Staffing • United States

Remote
USD 90,000 - 150,000
Software Research Analyst
Software Research Analyst

Questah Staffing • United States

Remote
USD 40,000 - 70,000
Scraping Engineer
Scraping Engineer

String • United States

Remote
USD 60,000 - 90,000
Data Engineer - Web Scraping
Data Engineer - Web Scraping

Alternative Path Private Limited • United States

Remote
USD 90,000 - 130,000
Remote AI Web Scraping Engineer: Stealth Data Pipelines
Remote AI Web Scraping Engineer: Stealth Data Pipelines

Workana LLC • United States

Remote
USD 140,000 - 210,000
Compensation in USD
Fully remote work
Career growth on an international公司
Member of Technical Staff | Knowledge Platform
Member of Technical Staff | Knowledge Platform

Avra • United States

Remote
USD 120,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Rachel Paul Recruiting • New York (NY)

Hybrid
USD 120,000 - 150,000
Experienced Web Scraping Engineer (Remote)
Experienced Web Scraping Engineer (Remote)

Source.ai • United States

Remote
USD 51,000 - 61,000
Web Scraping Specialist
Web Scraping Specialist

Wynd Labs • United States

Remote
USD 110,000 - 170,000
Opportunity
Culture
Compensation
Data Engineer - Web Scraping (3 months contract)
Data Engineer - Web Scraping (3 months contract)

CXG Holdings Inc • United States

Remote
USD 83,000 - 124,000