Software Engineer: Content Pipelines & Web Scraping

Jobgether; United States; Remote; USD 54,000 - 79,000; Full time

Benefits offered by this job

Remote work

Health coverage

RRSP matching

Life insurance

Employee assistance program

Paid time off

Holiday break

Meeting-free Fridays

Career development

Collaborative culture

Inclusive workplace

Job summary

Jobgether is seeking a data engineering-focused software engineer to build scalable web scrapers, content integration services, and Kafka-based pipelines. You will integrate external sources, develop Java-based backends, and ensure data quality in high-volume environments.

You will work remotely with a cross-functional team on architecture and performance optimizations, embracing innovation and advanced data processing techniques.

Qualifications

• Degree in Computer Science or related field, or equivalent practical experience.
• 3+ years of professional software development experience.
• Strong TypeScript or JavaScript skills for content integration, parsing, and data processing.
• Backend services development with Java or similar language.
• Production web scraping in production environments with responsible crawling practices.
• Experience with event-driven architectures, Kafka.
• Knowledge of HTML parsing, content extraction, data transformation, and normalization.
• High-volume, fault-tolerant data pipelines experience.
• SQL and data modeling knowledge.
• Linux, Git, Maven in modern workflows; Solr knowledge is a plus.
• Experience building content ingestion pipelines for news/media or data-intensive apps.
• Understanding distributed systems, scalability, reliability, and performance optimization.
• Ownership and initiative; can work independently.
• Analytical problem-solving and troubleshooting skills.
• Effective communication and teamwork.

Responsibilities

• Build scalable web scrapers and crawlers with responsible crawling.
• Develop content integration services using TypeScript/JavaScript.
• Design and maintain Kafka-based event-driven pipelines.
• Integrate external content sources, feeds, and APIs.
• Develop Java-based RESTful backend services.
• Improve data quality with validation and normalization practices.
• Monitor and optimize pipeline performance and costs.
• Contribute to system architecture and scalability decisions.
• Support search/indexing workflows with Solr integration.
• Collaborate with team and stakeholders; share knowledge.

Skills

TypeScript

JavaScript

Backend development

Kafka

Content processing

Data pipelines

SQL

Distributed systems

Problem solving

Communication

Teamwork

Innovation

Education

Bachelor’s degree in Computer Science or related field

Tools

Git

Maven

Linux

Solr

HTML parsing

Job description

Jobgether is seeking a data engineering-focused software engineer to build scalable web scrapers, content integration services, and Kafka-based pipelines. You will integrate external sources, develop Java-based backends, and ensure data quality in high-volume environments.

You will work remotely with a cross-functional team on architecture and performance optimizations, embracing innovation and advanced data processing techniques.