Scalable Web Crawling Engineer

Mindsite; Fatih; On-site; TRY 350,000 - 600,000; Full time

Benefits offered by this job

Competitive compensation

Flexible benefits

Additional leave

Culture of leadership

Professional development

Job summary

Mindsite is seeking a Reverse Engineer to design and build resilient web data acquisition systems. You will work across browser automation, network analysis, and scalable crawling infrastructure.

You’ll extract data from dynamic websites and analyze network traffic to identify endpoints. Collaborate with engineering and data teams to define schemas and delivery standards while improving system stability and scalability.

Qualifications

• 3+ years of experience in software engineering, reverse engineering, web scraping, crawling, or a similar technical role.
• Strong proficiency in Python and asynchronous programming.
• Advanced understanding of HTTP/HTTPS, sessions, cookies, headers, and request lifecycles.
• Hands-on experience with browser automation tools, especially Playwright or Selenium.
• Strong understanding of how modern websites work, including JavaScript-rendered content, XHR, GraphQL, and client-side applications.
• Experience analyzing network traffic and identifying relevant data endpoints using tools such as HAR files and browser developer tools.
• Experience building reliable data acquisition or crawling systems that operate at scale.
• Understanding of session management, rate limiting, retry strategies, and adaptive traffic management.

Responsibilities

• Design and develop robust, scalable web data acquisition and crawling systems.
• Extract structured data from dynamic and JavaScript-rendered websites.
• Analyze network traffic and application behavior to understand data delivery and identify endpoints.
• Build reliable mechanisms for retry, throttling, session management, and request handling.
• Investigate and resolve failures across network, browser, and application layers.
• Build and maintain session management systems supporting long-running data collection.
• Instrument and monitor crawl success rates, failure rates, and system performance.
• Analyze client-side behavior and request flows to understand backend communication.
• Define data schemas, collection requirements, and delivery standards with internal teams.
• Continuously improve stability, performance, and scalability of Mindsite’s data acquisition infrastructure.
• Contribute to the architecture of high-availability, production-grade crawling systems.

Skills

Python

Async programming

HTTP/HTTPS

Browser automation

Web scraping

GraphQL

HAR analysis

Tools

Playwright

Selenium

Job description

Mindsite is seeking a Reverse Engineer to design and build resilient web data acquisition systems. You will work across browser automation, network analysis, and scalable crawling infrastructure.

You’ll extract data from dynamic websites and analyze network traffic to identify endpoints. Collaborate with engineering and data teams to define schemas and delivery standards while improving system stability and scalability.