Data Engineer – WFH

Torentify; United States; Remote; USD 110,000 - 170,000; Full time

Benefits offered by this job

Equity

Remote-friendly

Job summary

Torentify is seeking an experienced Data Engineer to design and maintain data infrastructure for large-scale web datasets. You will work across Python code, web scraping, and distributed pipelines to ensure reliable data access and processing.

The role involves building scalable data pipelines, optimizing databases, and contributing to R&D efforts to improve data products, with remote-friendly workflows and a strong emphasis on production-grade engineering.

Qualifications

• Bachelor's degree or equivalent work experience.
• Strong Python programming skills for long-running data jobs.
• Hands-on experience with high-volume web scraping and anti-bot evasion.
• Experience building distributed data pipelines using Celery, Kafka, or RabbitMQ.
• Experience with columnar data warehouses and cost-aware cloud querying.
• Experience with Docker and Kubernetes deployments.
• Linux server troubleshooting and CI/CD for data workflows.

Responsibilities

• Maintain and optimize data systems to improve access and reliability.
• Develop scalable data pipelines for collecting, processing, and delivering large datasets.
• Build and maintain web scraping scripts for public data in line with company needs.
• Operate distributed workloads across multiple workers and servers.
• Create workflows for ingesting, segmenting, and annotating video and audio data.
• Monitor pipelines, fix data quality issues, and document processes.
• Contribute to R&D projects to improve data products and workflows.
• Support containerized workloads and scaling of scraping and processing systems.
• Develop scalable APIs and contribute to CI/CD for data engineering.

Skills

Advanced Python

Async programming

Multiprocessing

Web scraping at scale

Distributed data pipelines

Data warehousing

Docker

Kubernetes

CI/CD

APIs

Linux

Education

Bachelor’s degree or equivalent

Tools

Celery

Kafka

RabbitMQ

Databend

ClickHouse

BigQuery

Docker

Kubernetes

GitHub Actions

ArgoCD

Job description

About the Company

The company builds infrastructure for collecting, processing, and organizing large volumes of public web data used by companies developing advanced AI models. Its technology supports Grass, a bandwidth-sharing network that enables a distributed web crawler, along with data pipelines for ingesting, segmenting, and annotating billions of videos, transcripts, and audio files. The team focuses on expanding access to high-quality web data for AI development through scalable technical infrastructure.

About the Role

The role focuses on building and maintaining the data infrastructure required to collect, process, and deliver large-scale web datasets. You will work across Python development, web scraping, distributed data pipelines, data warehousing, and infrastructure management to support reliable data access and processing.

This position suits an experienced engineer who can write production-grade Python code, manage high-volume scraping workflows, and operate distributed systems across multiple workers and servers. You will also contribute to database optimization, pipeline troubleshooting, technical documentation, and research and development initiatives that improve the company’s data products.

Key Responsibilities

• Maintain, optimize, and troubleshoot database queries and data systems to improve data access, processing efficiency, and reliability.


• Develop and maintain scalable data pipelines for collecting, processing, transforming, validating, and delivering large datasets.


• Build, test, and maintain web scraping scripts and tools for collecting publicly available data in accordance with company requirements.


• Work with distributed data collection and processing workloads across multiple workers and servers.


• Develop and maintain workflows for ingesting, segmenting, and annotating large volumes of video, transcript, and audio data.


• Monitor pipeline performance, investigate data quality issues, and implement fixes to maintain accuracy and operational continuity.


• Document database queries, pipeline processes, scraping workflows, technical decisions, issues, and resolutions.


• Participate in research and development projects focused on improving data products and engineering workflows.


• Support containerized workloads, deployments, and scaling of scraping and data processing systems.


• Develop and maintain scalable APIs and contribute to CI/CD workflows for data engineering operations.

Required Qualifications

• Bachelor’s degree or equivalent work experience.


• Advanced Python programming skills, including asynchronous programming, multiprocessing, and production-grade code for long‑running data jobs.


• Hands‑on experience with high-volume web scraping, including proxies, rate limiting, and anti‑bot evasion techniques.


• Experience designing and operating distributed data pipelines using task queues such as Celery, Kafka, RabbitMQ, or similar technologies.


• Practical experience with columnar or analytical data warehouses, including complex analytical queries, partitioning strategies, and cost-aware cloud querying.


• Experience with Docker and Kubernetes, including containerizing workloads and managing deployments.


• Comfort managing Linux servers and troubleshooting performance issues involving disk I/O, networking, and memory.


• Experience with CI/CD for data workflows using tools such as GitHub Actions or ArgoCD.


• Experience writing scalable APIs.


• Ability to troubleshoot data systems, maintain data quality, and document technical work.

Preferred Qualifications / Requirements

• Experience with Databend, ClickHouse, or BigQuery is strongly preferred.


• Experience with platform APIs and large media or metadata datasets, including data from video platforms and social media.


• Experience creating Helm charts and Kubernetes manifests, as well as managing autoscaling for scraping and processing workloads.

Location and Work Arrangement

• The company operates as a fully remote team.

Salary & Benefits

• The company offers a competitive salary, benefits, and equity package. Specific compensation figures and individual benefit details are not provided in the listing.