Lead Software Engineer - Scraping

Next Ventures; New York (NY); Remote; USD 90,000 - 150,000; Full time

Job summary

An innovative FinTech company is seeking a Lead Software Engineer to spearhead web scraping initiatives. In this role, you will design, develop, and maintain scalable scraping solutions, ensuring they meet performance requirements. The ideal candidate is a seasoned Python developer with expertise in web scraping technologies like Scrapy and Beautiful Soup. You will lead a team, promote best practices, and drive innovation within the data technology team. This fully remote position offers a dynamic work environment and the opportunity to influence critical business decisions through data-driven insights.

Benefits

Comprehensive Benefits

Annual Discretionary Bonus

Qualifications

• Expert Python developer with extensive experience in web scraping and related technologies.
• Strong understanding of design patterns and SOLID principles.

Responsibilities

• Lead the design and development of scalable web scraping applications.
• Provide technical leadership and mentorship to the development team.

Skills

Python Development

Web Scraping

Technical Leadership

Problem-Solving

API Design

Continuous Improvement

Tools

Scrapy

Beautiful Soup

C#/.NET

AWS

GCP

Azure

JIRA

Confluence

GitHub

Job description

Role:

Our client is an innovative FinTech company and the market leader in providing data and business intelligence to help customers make faster and smarter investment decisions. They are looking for a Lead Software Engineer to drive and own their web scraping initiatives. This role will lead the design, development, and maintenance of scalable scraping solutions as part of their data technology team. The ideal candidate is an expert Python developer with strong experience in web scraping and related technologies.

Responsibilities

• Provide effective technical leadership and hands-on technical mentorship to developers.
• Lead the design and development of web scraping applications and services, ensuring they are scalable, efficient, and meet performance requirements.
• Drive technical decisions, ensuring that best practices are followed.
• Provide leadership in applying design patterns and adhering to SOLID engineering principles, with a focus on reusability, modularity, scalability, and fault tolerance.
• Utilize hands-on experience to design, build, and support web scraping solutions, particularly using Python and relevant libraries (e.g., Scrapy).
• Promote a culture of continuous improvement and high-quality code delivery.
• Partner with the broader technology leadership team to ensure service reliability through testing and release procedures.
• Stay current with industry trends in web scraping, data technologies, and emerging tools.
• Identify opportunities for innovation and process improvement.
• Ensure compliance with security and firmwide architecture standards.

Requirements

• Strong understanding of technical design patterns and SOLID engineering principles.
• Experience mentoring engineering teams, particularly in the context of data extraction and scraping projects.
• Expertise in Python development, particularly with web scraping libraries like Scrapy/Beautiful Soup.
• Proven experience in developing and maintaining large-scale web scraping solutions.
• Familiarity with C#/.NET technologies is a strong plus.
• Strong problem-solving skills and a proactive attitude.
• Strong understanding of API design principles and best practices.
• Experience with CI/CD pipelines and automated testing frameworks.
• Familiarity with cloud platforms (AWS/GCP/Azure).
• Experience with software development lifecycle tools such as JIRA, Confluence, GitHub, etc.
• Excellent communication and collaboration skills.

This position offers comprehensive benefits and is eligible for an annual discretionary bonus.

Fully remote candidates will be considered.

Cannot provide Visa sponsorship at this time.