Senior Python Developer: Web Scraping & Data Pipelines

Quality Ai; Northern (KY); On-site; USD 110,000 - 160,000; Full time

Benefits offered by this job

Healthcare from Day 1

Learning & Development platform (QCodr

International mobility opportunities

Job summary

QualityAI in the United States is seeking an experienced Python Developer specializing in web scraping and data extraction to join our data engineering team. You will design and maintain scalable pipelines that collect data from websites, APIs, and other sources to support analytics and investment research.

You should enjoy building resilient, well-tested systems, have strong knowledge of Scrapy, BeautifulSoup, Selenium/Playwright, and SQL/NoSQL databases, and collaborate with analysts to

Qualifications

• 5-8 years of Python development with 3+ years in scraping/data extraction.
• Proficient with Scrapy, BeautifulSoup, Requests/HTTPX, Selenium/Playwright.
• Experience with asynchronous/concurrent programming (asyncio, multithreading).
• Strong understanding of HTTP, HTML/CSS, DOM, and selectors.
• Experience with proxies, headless browsers, and anti-bot techniques.
• Data processing with Pandas/NumPy and regex.
• SQL and at least one NoSQL database.
• Experience building data pipelines (Airflow/Prefect).
• Git, CI/CD, Docker, and automated testing (pytest).
• Cloud platform experience (AWS/AZURE/GCP).
• Strong debugging and communication skills.

Responsibilities

• Design, develop, and maintain scalable web scraping pipelines in Python.
• Extract data from static pages, JS-heavy sites, and APIs.
• Handle anti-bot measures, rate limits, and dynamic content.
• Clean/validate/normalize data for analytics.
• Monitor scraper health, alerting, and quick fixes for data freshness.
• Optimize scrapers for speed, reliability, and cost.
• Store data in relational and NoSQL databases and cloud storage.
• Write clean, tested, documented code and participate in reviews.
• Collaborate with analysts and data scientists to meet data needs.
• Ensure compliance with terms, robots.txt, and data privacy.

Skills

Python

Web scraping

Scrapy

BeautifulSoup

REST/GraphQL APIs

Selenium

Playwright

Asyncio

SQL

NoSQL (MongoDB/PostgreSQL)

Airflow

Docker

CI/CD

Git

Cloud (AWS/GCP/Azure)

Education

Bachelor's degree in Computer Science, Engineering, Mathematics, Finance, or related field

Tools

Docker

Kubernetes

Airflow

Spark

Snowflake

BigQuery

PostgreSQL

Job description

QualityAI in the United States is seeking an experienced Python Developer specializing in web scraping and data extraction to join our data engineering team. You will design and maintain scalable pipelines that collect data from websites, APIs, and other sources to support analytics and investment research.

You should enjoy building resilient, well-tested systems, have strong knowledge of Scrapy, BeautifulSoup, Selenium/Playwright, and SQL/NoSQL databases, and collaborate with analysts to