Data Extraction Engineer: Web Scraping & Data Quality

Crossing Hurdles; United States; On-site; USD 70,000 - 120,000; Full time

Job summary

Crossing Hurdles is seeking a skilled data scraping engineer to design, develop, and maintain robust scraping scripts and apps across diverse online sources. You will extract data from static and dynamic sites, then clean, validate, and structure large datasets for analysis.

You will monitor pipelines, troubleshoot failures, and collaborate with teammates to align data deliverables with project goals, while documenting processes for scalability and future use.

Qualifications

• Proven experience building and maintaining web scrapers with Python, Scrapy or Selenium.
• Experience handling APIs, RESTful interfaces, and JSON/XML data.
• Strong data cleaning, manipulation, and storage using relational or NoSQL databases.
• Knowledge of web security concepts and best practices for ethical scraping.

Responsibilities

• Design, develop, and maintain robust web scraping scripts and applications to extract data from various online sources.
• Perform data extraction from multiple websites, effectively handling both static and dynamic content.
• Clean, validate, and structure large volumes of scraped data to ensure accuracy for further analysis.
• Monitor scraping pipelines for data quality, troubleshoot scraping failures, and implement solutions to maintain effectiveness.
• Collaborate with team members to define data requirements and deliverables, ensuring alignment with project goals.
• Document scraping processes and maintain code for scalability and reusability in future projects.

Skills

Python

Scrapy

Selenium

APIs

JSON/XML parsing

Data cleaning

Relational databases

NoSQL databases

Web security

Job description

Crossing Hurdles is seeking a skilled data scraping engineer to design, develop, and maintain robust scraping scripts and apps across diverse online sources. You will extract data from static and dynamic sites, then clean, validate, and structure large datasets for analysis.

You will monitor pipelines, troubleshoot failures, and collaborate with teammates to align data deliverables with project goals, while documenting processes for scalability and future use.