Senior Data Infrastructure Engineer: Scale Web Data

United States Digital Space LLC; United States; Remote; USD 140,000 - 220,000; Full time

Job summary

the company is seeking a Senior Software Engineer to join the Web Data team, building the next generation of our web crawling and data extraction infrastructure. This hands-on role will ship production code for large-scale systems and partner with product, data science, and leadership to deliver high impact.

You’ll work with Java and Python, run on GCP/AWS, and contribute to ETL/ELT pipelines, observability, and reliability.

Qualifications

• 5+ years of professional software engineering experience building production systems.
• Strong CS fundamentals: algorithms, data structures, concurrency, distributed systems.
• Proficiency in Java and/or Python.
• Track record of owning features end-to-end from design through deployment and operation.
• Comfortable making sound architectural decisions at the component level.

Responsibilities

• Design and implement components of scalable, fault-tolerant web crawling and extraction pipelines.
• Write clean, production-grade code in Java and Python.
• Build and operate ETL/ELT pipelines for large-scale data extraction and transformation.
• Work with cloud infrastructure on GCP and AWS, primarily on GKE.
• Improve observability, reliability, and operational excellence across the systems you contribute to.
• Partner with product and data science teams to deliver impactful solutions.
• Contribute to code reviews, documentation, and knowledge sharing across the team.
• Stay current with evolving web technologies, anti-crawling mechanisms, and AI-powered extraction approaches.

Skills

Java

Python

Distributed systems

Algorithms & data structures

CS fundamentals

Tools

BigQuery

Snowflake

Kubernetes (GKE/EKS)

Terraform

Apache Airflow

Kafka

Scrapy

Job description

the company is seeking a Senior Software Engineer to join the Web Data team, building the next generation of our web crawling and data extraction infrastructure. This hands-on role will ship production code for large-scale systems and partner with product, data science, and leadership to deliver high impact.

You’ll work with Java and Python, run on GCP/AWS, and contribute to ETL/ELT pipelines, observability, and reliability.