Web Scraping Data Engineer

Verition Fund Management LLC; New York (NY); On-site; USD 125,000 - 200,000; Full time

Job summary

A hedge fund company in New York is seeking a Data Engineer to develop and manage a Web Scraping service for internal clients. Candidates should have expertise in web scraping frameworks and data engineering practices, with 2-5 years of relevant experience. The role offers a salary range of $125,000 to $200,000 USD, depending on qualifications and experience.

Qualifications

• 2-5 years of experience in a similar role.
• Prior buy side experience is strongly preferred.
• Experience building data pipelines in AWS using various tools.

Responsibilities

• Utilize AI models to enable a scalable Web Scraping capability.
• Manage web scraping requests from intake to user access.
• Coordinate with Compliance on approvals.
• Implement data quality checks for scraped data.
• Promote best practices in data engineering.

Skills

Web Scraping frameworks

Python programming

Data pipeline orchestration tools

SQL and NoSQL databases

AWS cloud services

Communication skills

Data quality checks

Familiarity with DevOps practices

Time series data knowledge

Education

Bachelor’s degree in computer science, engineering, mathematics or related field

Tools

Scrapy

BeautifulSoup

Selenium

Playwright

AWS S3

AWS Lambda

Terraform

Job description

Company

Verition Fund Management LLC ("Verition") is a multi‑strategy, multi‑manager hedge fund founded in 2008. Verition focuses on global investment strategies including Global Credit, Global Convertible, Volatility & Capital Structure Arbitrage, Event‑Driven Investing, Equity Long/Short & Capital Markets Trading, and Global Quantitative Trading.

Data Engineer Role

We are seeking a Data Engineer with experience utilizing Web Scraping technologies, frameworks and libraries in addition to traditional data engineering practices. This role is new to the team, and we need an engineer who has experience building scrapes and interacting with Portfolio Managers on the buy side to augment our data team. Our Data team operates a cloud platform for structured data, market data, security master, streaming data, alternative data that may be acquired from APIs, files, websites, internal databases, etc. We want to enable a new service, Web Scraping, for our internal clients. Experience and knowledge of modern techniques for scraping, supporting large data sets, historical data, data monitoring, data validation, support, operations, request management, and interaction with clients such as Technology, Operations and Portfolio managers will be required. This role will be an opportunity for an aspiring data engineer to help establish a new process, learn and grow into a more senior data engineering role as the person shows a positive track record with our new scraping capability.

Responsibilities

• Utilize AI models, code, libraries or applications to enable a scalable Web Scraping capability
• Web Scraping Request Management including intake, assessment, accessing sites to scrape, utilizing tools to scrape, storage of scrape, validation and entitlement to users
• Fielding questions from users about the scrapes and websites
• Coordinating with Compliance on approvals and TOU reviews
• Experience building data pipelines in AWS platform utilizing existing tools like Cron, Glue, Eventbridge, Python based ETL, AWS Redshift
• Normalizing/standardizing vendor data, firm data for firm consumption
• Implement data quality checks to ensure reliability and accuracy of scraped data
• Coordinate with internal teams on delivery, access, requests, support
• Promote data engineering best practices

Required Skills and Qualifications

• Bachelor’s degree in computer science, engineering, mathematics or related field
• 2-5 years experience in a similar role
• Prior buy side experience is strongly preferred (multi‑strat/hedge funds)
• Capital markets experience is necessary with good working knowledge of reference data across asset classes and experience with trading systems
• AWS cloud experience with common services (S3, Lambda, Cron, Event Bridge, etc.)
• Experience with web scraping frameworks (Scrapy, BeautifulSoup, Selenium, Playwright, etc.)
• Strong hands‑on skills with NoSQL and SQL databases, programming in Python, data pipeline orchestration tools and analytics tools
• Familiarity with time series data and common market data sources (Bloomberg, Refinitiv, etc.) is a plus
• Familiarity with modern DevOps practices and infrastructure‑as‑code tools (e.g. Terraform, CloudFormation)
• Strong communication skills to work with stakeholders across technology, investment, and operations teams
• Ability to work independently or as part of a team in a fast‑paced environment with tight deadlines

Salary Range

$125,000 USD - $200,000 USD