Senior Web Scraping Engineer

Alternative Path; India; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Alternative Path is seeking skilled software developers to collaborate on client projects with an asset management firm. You will work with multiple teams, contribute ownership, and drive innovation for our platform.

The role emphasizes independence with strong engineering support and a meritocratic, team-first culture. As part of the Web Scraping Team, you will design and maintain web scrapers, perform data cleaning and transformation, and ensure high-quality datasets and timely notifications

Qualifications

• Bachelors/Masters in computer science or related field.
• 4-6 years of data engineering experience.
• Strong Python and SQL/Database skills.
• Expertise with Pandas for data cleaning and manipulation.
• Proficient in web scraping tools and techniques (Selenium, Scrappy, XPath).
• Experience with Airflow for workflow orchestration and data pipelines.
• Experience with large datasets, data cleaning, transformation, and replacements.
• Excellent verbal and written communication skills.
• Aptitude for designing data infrastructure and tools for data scientists.

Responsibilities

• Collaborate with analysts to understand requirements.
• Design, implement, and maintain web scrapers for diverse datasets.
• Perform data cleaning, transformation, and loading into databases/warehouse.
• Schedule scrapers and maintain data pipelines with Airflow or similar tools.
• Develop QC checks and alerting for time-sensitive data incidents.
• Contribute to building tools to enhance the web scraping platform.

Skills

Python

SQL

Data cleaning

Communication skills

Infrastructure design

Education

Bachelor's degree in computer science or related field

Tools

Pandas

Selenium

Scrappy

Fiddler

XPath

Airflow

Docker

Kubernetes

Jenkins

GitHub Actions

CI/CD

AWS (S3, RDS, SNS, SQS, Lambda)

Job description

Alternative Path is seeking skilled software developers to collaborate on client projects with an asset management firm. In this role, you will collaborate with individuals across various company departments to shape and innovate new products and features for our platform, enhancing existing ones. You will have a large degree of independence and trust, but you won't be isolated; the support of the Engineering team leads, the Product team leads, and every other technology team member is behind you. This is an opportunity to join a team-first meritocracy and help grow an entrepreneurial group inside Alternative Path. You will be asked to contribute, given ownership, and expected to make your voice heard.

Role Summary

Perform web scraping using various scraping techniques and then utilize Python’s Pandas library for data cleaning and manipulation. Then ingest the data into a Database/Warehouse and schedule the scrapers using Airflow or other tools.

Requirements

Role Overview

The Web Scraping Team at Alternative Path is seeking a creative and detail-oriented developer to contribute to client projects. The team develops essential applications, datasets, and alerts for various teams within the client's organization, supporting their daily investment decisions. The mission is to maintain operational excellence by delivering high-quality proprietary datasets, timely notifications, and exceptional service. We are seeking someone self-motivated, self-sufficient, with a passion for tinkering and a love for automation.

In your role, you will:

• Collaborate with analysts to understand and anticipate requirements.
• Design, implement, and maintain web scrapers for a wide variety of alternative datasets.
• Perform data cleaning, Exploration, transformation, etc., of scraped data.
• Collaborate with cross-functional teams to understand data requirements and implement efficient data processing workflows.
• Author QC checks to validate data availability and integrity. Maintain alerting systems and investigate time-sensitive data incidents to ensure smooth day- today operations.
• Design and implement products and tools to enhance the Web scraping Platform.

Qualifications

Must have

• Bachelors/master’s degree in computer science or in any related field
• 4-6 years of experience in the data engineering field
• Strong Python and SQL/Database skills
• Strong expertise in using the Pandas library (Python) is a must
• Strong expertise with scraping and common scraping tools(Selenium, Scrappy, Fiddler, Postman, XPath)
• Strong expertise in using the Apache Airflow platform for workflow orchestration and data pipeline management is a must.
• Experience with web technologies(HTML/JS, APIs, etc.)
• Proven work experience in working with large data sets for Data cleaning, Data transformation, Data manipulation, and Data replacements.
• Excellent verbal and written communication skills
• Aptitude for designing infrastructure, data products, and tools for Data Scientists

Preferred

• Experience containerizing workloads with Docker(Kubernetes a plus)
• Experience with building automation (Jenkins, GitHub Actions, CI/CD)
• Experience with AWS technologies like S3, RDS, SNS, SQS, Lambda, etc