Data Engineer: Scalable Web Data & Pipelines (Remote)

Wynd Labs; New York (NY); Remote; USD 140,000 - 190,000; Full time

Benefits offered by this job

Equity package

Fully remote team

Job summary

Wynd Labs in the United States is seeking an experienced data engineer to build and optimize large-scale web data pipelines for AI model training. You will work on high-volume scraping, distributed processing, and analytics workloads, ensuring scalable, reliable data delivery across many workers and servers.

The role emphasizes Python, async programming, Docker & Kubernetes, and CI/CD for data workflows. You’ll contribute to platform APIs and data warehousing with modern cloud warehouses, in a

Qualifications

• Bachelor’s degree or equivalent work experience.
• Strong Python (advanced) and async programming experience.
• Experience with high-volume web scraping and anti-bot evasion.
• Experience with distributed data pipelines and task queues (Celery, Kafka, RabbitMQ).
• Background in data warehousing and SQL (BigQuery/ClickHouse).
• Experience containerizing workloads with Docker and Kubernetes.
• Familiarity with CI/CD for data workflows (GitHub Actions, ArgoCD).
• Ability to design scalable APIs and data platforms.

Responsibilities

• Maintain and optimize data pipelines for large-scale datasets.
• Support web scraping and data collection initiatives.
• Monitor data quality and troubleshoot pipeline issues.
• Document engineering decisions and pipeline workflows.
• Collaborate on R&D projects to improve data products.

Skills

Python

Async programming

Web scraping

High-volume scraping

Distributed data pipelines

Task queues

Data warehousing

Databend

ClickHouse

BigQuery

Docker

Kubernetes

Helm charts

CI/CD

GitHub Actions

ArgoCD

Linux administration

Performance debugging

API design

Scalable APIs

Education

Bachelor’s degree or equivalent

Tools

Docker

Kubernetes

Helm

Job description

Wynd Labs in the United States is seeking an experienced data engineer to build and optimize large-scale web data pipelines for AI model training. You will work on high-volume scraping, distributed processing, and analytics workloads, ensuring scalable, reliable data delivery across many workers and servers.

The role emphasizes Python, async programming, Docker & Kubernetes, and CI/CD for data workflows. You’ll contribute to platform APIs and data warehousing with modern cloud warehouses, in a