Director of Software Engineering (Node.js & Web Scraping Expert)

Roman Health Pharmacy LLC; Los Angeles (CA); On-site; USD 180,000 - 220,000; Full time

Job summary

A leading company in web scraping technology is seeking a Director of Software Engineering to lead their engineering team. This position entails designing and optimizing high-performance global web scraping systems, ensuring compliance with legal standards, and fostering innovation within the team. The ideal candidate will possess extensive experience in handling anti-bot measures and cloud-based architecture, bringing a track record of mentoring and scaling technical teams.

Qualifications

• 10+ years of experience in software engineering, 5+ years in web scraping.
• Hands-on expertise in Node.js and headless browser automation.
• Experience with CI/CD and cloud deployment.

Responsibilities

• Lead the engineering team, architecting scalable web scraping systems.
• Design data extraction pipelines for structured and unstructured data.
• Implement best practices for microservices and API integrations.

Skills

Problem-solving

Debugging

Leadership

Communication

Tools

Node.js

Puppeteer

Playwright

Selenium

Kafka

Docker

Kubernetes

AWS

GCP

Azure

Job description

We are seeking a Director of Software Engineering with deep expertise in Node.js development and large-scale web scraping. This role will lead the engineering team, designing and optimizing high-performance, distributed web scraping systems. The ideal candidate has extensive experience in handling anti-bot measures, data pipeline optimization, and scalable cloud-based architectures.

Key Responsibilities - Software Engineering & Web Scraping Leadership:

• Architect, develop, and maintain scalable and distributed web scraping systems using Node.js.
• Design and implement data extraction pipelines to process large volumes of structured and unstructured data.
• Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation.
• Optimize scraping processes for performance, reliability, and efficiency while managing proxy services (residential, datacenter, rotating). Oversee data storage and processing strategies, ensuring high availability and consistency.
• Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications.
• Implement best practices for microservices, API integrations, and real-time data streaming.

Key Responsibilities - Scalability, Security & DevOps:

• Lead the transition to cloud-native, containerized, and serverless architectures for web scraping.
• Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.).
• Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping.
• Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks.
• Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability.

Key Responsibilities - Engineering Team Management & Strategy:

• Lead, mentor, and grow a high-performance engineering team.
• Define and execute the technology roadmap, aligning with business objectives.
• Foster a culture of continuous learning, collaboration, and innovation.
• Implement agile development methodologies (Scrum, Kanban) to optimize project execution.
• Ensure code quality, security, and best practices across all engineering efforts.

Qualifications & Experience - Technical Expertise:

• 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
• Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
• Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
• Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs). Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
• Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.). Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).

Qualifications & Experience - Leadership & Soft Skills:

• Proven track record of scaling engineering teams and leading complex projects.
• Strong problem-solving and debugging skills, especially for scraping challenges and performance bottlenecks.
• Excellent communication and stakeholder management skills.
• Passion for mentorship, team development, and continuous learning.

Preferred Qualifications:

• Experience with machine learning for data extraction and NLP.
• Knowledge of browser fingerprinting and bot detection mechanisms.
• Familiarity with enterprise-scale web crawling frameworks (Scrapy, Colly, Apify, etc.).
• Prior leadership experience in data-driven businesses or web scraping startups.