Lead Data Engineer – Web Crawling & Large-Scale Data

Virtus Lab sp. z o.o. (Ltd.); New York (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

A technology consultancy in New York is seeking an experienced Technical Leader to build and manage large-scale data systems. The role demands deep expertise in web crawling, data engineering, and cloud environments. You will mentor engineers, engage with clients to analyze requirements, and lead architecture discussions. The ideal candidate has strong background in building scalable systems and is passionate about technical leadership. The position offers a collaborative work environment focused on innovation.

Qualifications

• Proven experience building web crawling or large-scale data systems from scratch.
• Strong architectural skills in designing scalable, fault-tolerant distributed systems.
• Deep background in large-scale data engineering handling terabytes daily.

Responsibilities

• Support U.S. and international teams with technical expertise for client activities.
• Conduct requirements analysis and solution discovery to align business needs.
• Prepare technical sections of commercial offers including solution descriptions.

Skills

Web crawling technologies

Data engineering

Technical mentorship

Architectural design

Cloud data warehouses

Apache Kafka

Kubernetes

ETL/ELT pipelines

Requirements analysis

Client engagement

Tools

BigQuery

Snowflake

Scrapy

Airflow

Job description

A technology consultancy in New York is seeking an experienced Technical Leader to build and manage large-scale data systems. The role demands deep expertise in web crawling, data engineering, and cloud environments. You will mentor engineers, engage with clients to analyze requirements, and lead architecture discussions. The ideal candidate has strong background in building scalable systems and is passionate about technical leadership. The position offers a collaborative work environment focused on innovation.