Senior Data Engineer: ETL, Web Scraping, & Cloud DB

WebDataGuru; Indiana (PA); On-site; USD 110,000 - 170,000; Full time

Job summary

WebDataGuru is seeking a Senior Data Engineer to lead ETL development and large-scale data extraction initiatives. The role focuses on building robust data pipelines, implementing web scraping strategies, and delivering scalable data solutions across cloud platforms.

You will optimize ETL processes, manage data warehouses, and collaborate cross-functionally to ensure data quality from source to analytics. A strong emphasis on security, performance, and CI/CD is required.

Qualifications

• ETL pipeline development and optimization for large datasets.
• Bulk Data Extraction using web scraping or crawling techniques.
• Parallelism, concurrency, and threading to improve performance.
• Python-based libraries: Numpy, Pandas, Selenium, Scrapy.
• Postgres, MySQL, and NoSQL databases.
• AWS, GCP, or Azure with Lambda, EC2, S3.
• REST APIs development and integration.
• SQL procedures, complex queries, and data pipelines.
• Git and SVN version control.
• SDLC, testing methodologies, and CI/CD processes.
• Strong analytical and problem-solving abilities for scalable data solutions.

Responsibilities

• ETL Pipeline Management across production, QA, and UAT.
• Develop, optimize, and troubleshoot ETL pipelines for large datasets.
• Handle large datasets and manage server infrastructure.
• Web Scraping and Crawling to extract data from hundreds of websites.
• Apply parallelism/concurrency/ threading to improve extraction performance.
• Leverage Selenium and Scrapy to build scalable scraping solutions.
• Work with Postgres, MySQL, and NoSQL databases.
• Implement cloud best practices with AWS, GCP, or Azure (Lambda/EC2/S3).
• Create and optimize SQL procedures and data pipelines for integration.
• Collaborate with teams to deliver end-to-end data solutions.
• Ensure data quality and proper mappings across environments.

Skills

ETL Development

Bulk Data Extraction

Parallel Processing

REST API

SQL Proficiency

Version Control

SDLC Knowledge

Problem-Solving Skills

Tools

Selenium

Scrapy

Pandas

NumPy

Requests

xpath

Urllib

lxml

Postgres

MySQL

NoSQL

AWS

GCP

Azure

Lambda

EC2

S3

Job description

WebDataGuru is seeking a Senior Data Engineer to lead ETL development and large-scale data extraction initiatives. The role focuses on building robust data pipelines, implementing web scraping strategies, and delivering scalable data solutions across cloud platforms.

You will optimize ETL processes, manage data warehouses, and collaborate cross-functionally to ensure data quality from source to analytics. A strong emphasis on security, performance, and CI/CD is required.