Automation Engineer - Worlwide Remote

Invisible Technologies; San Francisco (CA); Remote; USD 80,000 - 120,000; Full time

Job summary

Invisible Technologies in San Francisco is looking for a skilled Python Developer to create and maintain web scrapers for various platforms. The ideal candidate will have extensive experience developing Python scripts with a focus on overcoming technical challenges in web scraping.

The position requires a solid background in API development, SQL databases, and familiarity with various web development frameworks. Strong interpersonal skills and the ability to work with diverse teams are essential for this position.

Qualifications

• 3+ years of experience in Python programming and web scraping.
• Strong SQL knowledge with experience in RDBMS.
• Familiarity with containerization using Docker.

Responsibilities

• Create crawlers using Python and manage web scraping tasks.
• Develop and maintain data pipelines for crawler output.
• Handle technical challenges like IP bans and CAPTCHAs.

Skills

Python programming

Web scraping (Requests, BeautifulSoup, Selenium, Scrapy)

APIs and REST

SQL and RDBMS

Object-oriented programming

Version control (Git)

Tools

Scrapy

Selenium

Docker

Flask

Django

FastAPI

Airflow

Job description

Job Description

• Create crawlers for each source URL using Python modules (Scrapy, Selenium, Requests, BeautifulSoup, Splash).
• Develop and maintain Scrapy pipelines and middlewares to manage crawler output.
• Build crawlers for all types of websites, overcoming technical roadblocks.
• Manage crawlers to handle IP bans, geolocation bans, CAPTCHAs, and bot‑blocking services.
• Write SQL queries for database operations using Python modules such as SQLAlchemy.
• Deploy the Python scripts and crawlers to Linux‑based AWS servers.

Skills Must Haves

• Strong hands‑on experience in Python programming.
• Good experience with scraping libraries such as Requests, BeautifulSoup, Selenium and Scrapy.
• Proven 3+ years working with web development frameworks such as Flask, Django, FastAPI, Tornado, and pandas, and building APIs and services using REST.
• Experience with any RDBMS and strong SQL knowledge.
• Clear understanding of object‑oriented concepts.
• Proficient understanding of code versioning tools like Git.

Nice to Have

• Familiarity with UI frameworks such as Angular or ReactJS.
• Strong unit‑testing and debugging skills.
• Knowledge of data engineering platforms such as Airflow.
• Experience with Docker.
• Knowledge of Agile and Scrum practices; experience with Jira and Confluence is a plus.
• Experience with ETL and Celery.
• Excellent interpersonal skills and ability to work with a diverse team.

Qualifications

• Python
• Web Scraping using Requests, BeautifulSoup, Selenium, and Scrapy
• Frameworks such as Flask, Django, FastAPI, Tornado, pandas
• Building APIs and services using REST