Python Web Scraping Engineer — Crawler Architect

Invisible Technologies; San Francisco (CA); Remote; USD 80,000 - 120,000; Full time

Job summary

Invisible Technologies in San Francisco is looking for a skilled Python Developer to create and maintain web scrapers for various platforms. The ideal candidate will have extensive experience developing Python scripts with a focus on overcoming technical challenges in web scraping.

The position requires a solid background in API development, SQL databases, and familiarity with various web development frameworks. Strong interpersonal skills and the ability to work with diverse teams are essential for this position.

Qualifications

• 3+ years of experience in Python programming and web scraping.
• Strong SQL knowledge with experience in RDBMS.
• Familiarity with containerization using Docker.

Responsibilities

• Create crawlers using Python and manage web scraping tasks.
• Develop and maintain data pipelines for crawler output.
• Handle technical challenges like IP bans and CAPTCHAs.

Skills

Python programming

Web scraping (Requests, BeautifulSoup, Selenium, Scrapy)

APIs and REST

SQL and RDBMS

Object-oriented programming

Version control (Git)

Tools

Scrapy

Selenium

Docker

Flask

Django

FastAPI

Airflow

Job description

Invisible Technologies in San Francisco is looking for a skilled Python Developer to create and maintain web scrapers for various platforms. The ideal candidate will have extensive experience developing Python scripts with a focus on overcoming technical challenges in web scraping.

The position requires a solid background in API development, SQL databases, and familiarity with various web development frameworks. Strong interpersonal skills and the ability to work with diverse teams are essential for this position.