Senior Data Engineer: Big Data & Web Scraping Pipelines

Vantage Point Global; Warszawa; On-site; PLN 180,000 - 280,000; Full time

Job summary

Vantage Point Global is seeking an experienced Data Engineer to architect and maintain high-throughput data pipelines in Databricks and AWS. You will ingest, normalize, and enrich data from diverse sources, collaborating with AI/ML teams to translate requirements into scalable architectures and APIs.

You will optimize performance, enforce governance, and contribute to vendor onboarding while evaluating GenAI tooling and compliant scraping workflows. Strong English and ethics are essential.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering or related role.
• Strong Python and SQL skills.
• Experience with Spark or Scala and distributed data processing.
• Proficiency in building scalable, distributed data pipelines in a cloud environment.
• Familiarity with Linux/UNIX, HTTP, HTML, JavaScript, and networking concepts.
• Knowledge of web scraping tools and libraries (Requests, BeautifulSoup, Scrapy, Pandas, Selenium, Spark).
• Version control systems and open-source practices.
• Solid understanding of data architecture, data modeling, and data warehousing.
• Excellent analytical and problem-solving skills; strong English written and spoken.
• Commitment to high ethical standards; bonus for PDFs/images extraction.
• Experience with system monitoring/administration tools; graph databases; large-scale data analysis.

Responsibilities

• Architect, develop, and maintain high-throughput data pipelines in Databricks and AWS.
• Ingest, normalize, and enrich large volumes of structured and unstructured data from multiple sources.
• Collaborate with AI Engineers, ML scientists, and software teams to design scalable data architectures, schemas, and APIs.
• Optimize pipeline performance and cost using distributed processing and AWS best practices.
• Enforce data governance, privacy, and lineage standards.
• Catalogue assets in Unity Catalog and manage PII/PCI classifications.
• Build automated validation, testing, and monitoring for data quality and freshness.
• Support onboarding and integration of external data vendors; ensure compliance and value.
• Evaluate GenAI tooling and drive proof-of-concepts; explore scraping tools and workflows.
• Own tools for compliant web crawling and scraping; test and validate scraped data for quality.
• Identify and resolve scraping issues; scale processes as needed.

Skills

Python

SQL

Spark/Scala

Cloud

Linux/UNIX

Web scraping

Git

Data warehousing

Data modeling

English

Education

Bachelor’s or Master’s degree in CS/Engineering

Tools

Requests

BeautifulSoup

Scrapy

Pandas

Selenium

Spark

Job description

Vantage Point Global is seeking an experienced Data Engineer to architect and maintain high-throughput data pipelines in Databricks and AWS. You will ingest, normalize, and enrich data from diverse sources, collaborating with AI/ML teams to translate requirements into scalable architectures and APIs.

You will optimize performance, enforce governance, and contribute to vendor onboarding while evaluating GenAI tooling and compliant scraping workflows. Strong English and ethics are essential.