ETL/DB/Data Extraction Engineer

WebDataGuru; Indiana; On-site; USD 120,000 - 170,000; Full time

Job summary

WebDataGuru is seeking an exceptionally skilled Senior Data Engineer to design, build, and optimize ETL pipelines, and to drive bulk data extraction via web scraping and crawling techniques. You will manage large-scale data projects with emphasis on security, scalability, and performance across cloud environments.

You will collaborate with cross-functional teams, implement efficient data solutions, and ensure data quality while leveraging Python libraries and frameworks like Scrapy and Selenium

Qualifications

• ETL pipeline development and optimization for large datasets.

Responsibilities

• Develop, optimize, and troubleshoot ETL pipelines across environments.

Skills

ETL Development

Bulk Data Extraction

Parallel Processing

Python Data Stack

Database Management

Cloud Technologies

REST API

SQL Proficiency

Version Control

SDLC & CI/CD

Problem Solving

Tools

Selenium

Scrapy

Requests

XPath

urllib

lxml

Job description

WebDataGuru is hiring a ETL/DB/Data Extraction Engineer

10.10.24

We are seeking an exceptionally skilled Senior Data Engineer with extensive experience in ETL development and Bulk Data Extraction through Web Scraping and Crawling technologies. The ideal candidate will have a solid background in building, optimizing, and maintaining ETL pipelines, with strong expertise in data management and cloud technologies. This is a challenging role designed for a detail-oriented professional who is passionate about delivering high-quality data solutions.

You will be responsible for managing large-scale data projects, ensuring best practices in security, scalability, and performance. If you have a proven track record of delivering comprehensive data engineering solutions across diverse industries, we would love to hear from you.

Roles & Responsibilities

• ETL Pipeline Management:
• Develop, optimize, and troubleshoot ETL pipelines across production, QA, and UAT environments.
• Handle large datasets and manage server infrastructure effectively.
• Web Scraping and Crawling:
• Implement advanced web scraping techniques for rapid and efficient data extraction from large-scale sources across hundreds of websites.
• Performance Optimization:
• Apply techniques such as parallelism, concurrency, and threading to improve data extraction and processing efficiency.
• Library and Framework Expertise:
• Utilize relevant tools such as requests, xpath, ActionChain, Urllib, Numpy, Pandas, and lxml.
• Work with frameworks like Selenium and Scrapy to build scalable scraping solutions.
• Database Management:
• Work with both relational databases (Postgres, MySQL) and NoSQL databases.
• Optimize database performance, including schemas, triggers, and queries.
• Cloud and API Integration:
• Implement cloud best practices for security, scalability, and performance using AWS, GCP, or Azure.
• Utilize AWS services such as Lambda, EC2, and S3 alongside external APIs.
• Data Engineering:
• Participate in data warehousing, architecture, and ETL pipeline development at an enterprise scale.
• Create and optimize SQL procedures and data pipelines for seamless integration.
• Collaboration and Project Delivery:
• Work cross-functionally with teams to deliver end-to-end data solutions.
• Ensure source data quality, develop mapping and transformations, and validate jobs across various environments.

Skills & Requirements

• ETL Development Expertise: Proficiency in ETL pipeline development and optimization for large datasets.
• Bulk Data Extraction: Experience with Web Scraping and Crawling technologies.
• Parallel Processing: Strong understanding of parallelism, concurrency, and threading.
• Technical Stack: Expertise in Python-based libraries like Numpy, Pandas, Selenium, and Scrapy.
• Database Management: Proficient in Postgres, MySQL, and NoSQL databases.
• Cloud Technologies: Experience in AWS, GCP, and Azure, with hands-on knowledge of AWS Lambda, EC2, and S3.
• REST API: Experience with developing and integrating REST APIs.
• SQL Proficiency: Skilled in SQL procedures, query optimization, and pipeline creation.
• Version Control: Familiarity with code versioning tools like Git and SVN.
• SDLC Knowledge: Solid understanding of the Software Development Life Cycle (SDLC), testing methodologies, and CI/CD processes.
• Problem-Solving Skills: Exceptional analytical and problem-solving abilities with a strong focus on delivering scalable data solutions.

Experience

• Minimum 4-5 years of experience in ETL development, including experience with Bulk Data Extraction through web scraping and crawling technologies.
• Proven track record of delivering large-scale data engineering solutions.
• Prior experience with cloud environments like AWS, GCP, or Azure is highly preferred.

Why Join Us?

• Opportunity to work on high-impact, large-scale data projects.
• Collaborative, growth-oriented environment where innovation is encouraged.
• Competitive compensation and benefits package.