Python Developer (Web Scraper)

PT BCI CENTRAL INDONESIA; Jakarta Utara; On-site; IDR 200.000.000 - 300.000.000; Full time

Job summary

A technology firm is looking for a Senior Python Developer to lead their Web Harvesting team. In this role, you will be responsible for building and enhancing automated web scraping solutions, mentoring junior developers, and architecting scalable AWS solutions. The ideal candidate will have over 7 years of experience in software engineering, with strong skills in Python and web scraping. Benefits include medical allowance and sports facilities.

Benefits

Medical

Miscellaneous allowance

Gym

Parking

Vision

Annual Leave

Birthday Leave

Charity Day Leave

Qualifications

• 7+ years of experience in software engineering, focusing on Python.
• Proven leadership experience in mentoring junior engineers.
• Strong understanding of web scraping and handling complex document structures.

Responsibilities

• Lead the design and optimization of Python web harvesting pipelines.
• Collaborate with AI developers to improve JSON configuration.
• Architect scalable solutions in AWS cloud environments.

Skills

Python

Leadership/Mentorship

Web Scraping

AWS Services

Postgres

REST API Design

Tools

Airflow

AWS

Job description

We are seeking a highly skilled senior python developer to drive innovation and delivery within our Web Harvesting team. This role will lead efforts to build, scale, and enhance automated web scraping solutions powered by AI-assisted configuration generation. You will mentor junior developers, accelerate delivery timelines, and help architect solutions that handle increasingly complex data sources.

Our systems ingest data from multiple content types while ensuring AI is used responsibly – only to generate scraper configurations, not to extract the data itself. These configurations are transformed into Python-driven pipelines that power our harvesting at scale.

Key Responsibilities

• Lead the design, development, optimization and technical direction of Python-based web harvesting pipelines.
• Collaborate with AI agent developers to improve JSON configuration generation and integrate them seamlessly into harvesting workflows.
• Extend and enhance configuration frameworks to handle more complex and dynamic web sources.
• Design and oversee ingestion strategies for structured and unstructured offline files
• Architect scalable solutions in AWS cloud environments, leveraging Airflow, S3, Postgres, and API-driven services.
• Ensure best practices in data quality, reliability, and maintainability of harvesting pipelines.
• Partner with product and data teams to align harvesting output with business needs.

Required Skills & Experience

• 7+ years of experience in software engineering, with strong expertise in Python.
• Proven leadership/mentorship experience guiding and developing junior engineers.
• Deep understanding of web scraping, HTML parsing, and handling complex document structures (PDF, DOCx, CSV).
• Strong experience with AWS services (S3, ECS, Lambda, IAM, etc.) and workflow orchestration tools (preferably Airflow).
• Proficiency with relational databases (Postgres) and building robust data ingestion pipelines.
• Knowledge of REST API design and integration.
• Ability to architect systems for scale, resilience, and performance.

Preferred Skills

• Experience working with AI/ML systems for automation (e.g., using AI for config or workflow generation).
• Familiarity with distributed systems and cloud-native design patterns.
• Strong background in software engineering best practices (CI/CD, testing, code reviews).

About the Company

Hubexo provides cutting-edge data, insights, and software solutions to the global construction industry. Founded in Sweden in 1936, Hubexo specializes in project information, eTendering, product information, market intelligence, and specification.

With operations in more than 20 countries and a workforce of 2,500 employees, Hubexo helps its customers sell more efficiently, build sustainably, and lead the future of construction innovation.

Benefits

• Medical
• Miscellaneous allowance
• Sports (e.g. Gym)
• Parking
• Vision
• Annual Leave, Birthday Leave, Charity Day Leave, B