Software Developer (Content Integration & Data Pipelines)

Remote; San Francisco (CA); Remote; USD 54,000 - 79,000; Full time

Benefits offered by this job

Remote work

Health benefits

RRSP matching

Life insurance

EAP

Paid vacation

Week off Christmas–New Year

No meetings Fridays

Career development

Team collaboration

Inclusive workplace

Job summary

Remote is seeking a Software Developer (Content Integration & Data Pipelines) to build production-grade data pipelines that collect, process, enrich, and deliver online news content. You will connect thousands of sources to search, analytics, and AI-powered capabilities using TypeScript, Java, Kafka, and Solr.

The role spans web scraping, content integration, backend services, and event-driven processing to ensure reliable, scalable ingestion while adhering to responsible crawling practices.

Qualifications

• Degree in Computer Science or related field or equivalent practical experience.

Responsibilities

• Build scalable web scrapers and crawlers with responsible crawling practices.
• Develop content integration services for parsing, extraction, and data normalization.
• Design event-driven pipelines using Kafka for near real-time processing.
• Integrate external content sources via RSS, Atom feeds and APIs.
• Develop Java-based RESTful services for content processing and delivery.
• Improve data quality, validation, and fault-tolerant pipeline operations.
• Monitor and optimize pipeline performance and observability.

Skills

TypeScript

JavaScript

Problem solving

Analytical thinking

Content processing

Data pipelines

Distributed systems

Communication

Education

Bachelor's degree in Computer Science or related field

Tools

Kafka

Solr

Java

Maven

Git

Linux

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Developer (Content Integration & Data Pipelines) based in Canada.

We are seeking a skilled Software Developer to build and scale production-grade data pipelines that collect, process, enrich, and deliver online news and web content across a sophisticated media intelligence platform. In this role, you will help connect thousands of content sources to search, analytics, and AI-powered capabilities used by communications professionals. You will work across web scraping, content integration, backend development, and event-driven processing to ensure information is delivered accurately and efficiently. Using technologies such as TypeScript, Java, Kafka, and Solr, you will contribute to the architecture and evolution of high-volume content ingestion systems. You will play an active role in improving pipeline reliability, performance, observability, and data quality while following responsible web crawling practices. This remote opportunity offers the chance to solve complex engineering challenges, influence technical direction, and collaborate with a team focused on innovation, automation, and scalable data solutions.

Accountabilities

• Build scalable web scrapers and crawlers: Develop and maintain production-ready scraping and crawling solutions that collect online content responsibly, following standards such as robots.txt, rate limiting, and applicable terms of use.
• Develop content integration services: Implement parsing, extraction, and normalization logic using TypeScript and JavaScript to transform content from diverse sources into consistent, usable data.
• Design event-driven pipelines: Build and maintain Kafka-based services that process, enrich, and route content in near real time, supporting reliable and scalable data flows.
• Integrate external content sources: Connect third-party content providers, RSS and Atom feeds, and external APIs to expand and maintain content coverage.
• Develop backend services: Design and implement Java-based RESTful services that support content processing, integration, and delivery across the platform.
• Improve data quality and reliability: Establish and apply best practices for error handling, source management, validation, data normalization, and fault-tolerant pipeline operations.
• Monitor and optimize pipeline performance: Implement observability and monitoring practices, investigate processing issues, and improve system performance, reliability, and operating costs.
• Contribute to technical architecture: Participate in system design discussions, evaluate scalability trade-offs, and help define the technical direction of content acquisition and data processing systems.
• Support search and indexing capabilities: Contribute to content indexing and retrieval workflows, collaborating with search technologies such as Solr to ensure content is structured and accessible.
• Collaborate across the development team: Work with technical colleagues and other stakeholders to deliver maintainable solutions aligned with product and business needs.
• Mentor and share knowledge: Support junior developers, contribute to code reviews and engineering best practices, and promote continuous learning within the team.

Requirements:

• Education and experience: A degree in Computer Science or a related field, or equivalent practical experience, with at least three years of professional software development experience.
• TypeScript and JavaScript proficiency: Strong programming skills in TypeScript or JavaScript, particularly for content integration, parsing, and data processing.
• Backend development experience: Solid experience building backend services with Java or a comparable programming language.
• Production web scraping expertise: Hands-on experience developing and operating web scrapers and crawlers in production environments, with an understanding of robots.txt, rate limiting, responsible crawling practices, and applicable terms of use.
• Event-driven architecture: Practical experience with event-driven systems and message-streaming technologies, particularly Kafka.
• Content processing skills: Knowledge of HTML parsing, content extraction, data transformation, and normalization techniques.
• Data pipeline engineering: Experience designing or maintaining high-volume, fault-tolerant data pipelines that process and deliver information reliably.
• Database and data modeling knowledge: Familiarity with SQL and data modeling principles.
• Development tools and environments: Experience working in Linux environments and using Git and modern software development workflows. Familiarity with Maven is also expected.
• Search technology knowledge: Experience with Solr, including indexing, schema design, and query tuning, is an asset.
• Relevant industry experience: Experience building content ingestion pipelines for news, media, or similar data-intensive applications is strongly preferred.
• System design capabilities: Understanding of distributed systems, scalability trade-offs, reliability, and performance optimization.
• Ownership and initiative: Ability to work independently, take responsibility for projects, and proactively identify opportunities for technical improvement.
• Analytical problem-solving: Strong troubleshooting skills and the ability to investigate complex technical issues and develop practical solutions.
• Communication and teamwork: Ability to communicate effectively with technical and non-technical stakeholders, collaborate across teams, and share knowledge.
• Interest in innovation: Enthusiasm for data engineering, automation, emerging technologies, and AI-powered capabilities.

Benefits:

• Competitive compensation: Expected annual salary range of CAD $75,000-$110,000, depending on experience, skills, and internal equity.
• Remote work: Fully remote working environment.
• Health coverage: Health, dental, and vision benefits.
• Retirement savings: RRSP matching.
• Financial protection: Life insurance.
• Employee support: Access to an Employee Assistance Program (EAP).
• Paid time off: Paid vacation, personal days, and sick days.
• Holiday break: A week off between Christmas and New Year's.
• Meeting-free Fridays: No internal meetings scheduled on Fridays.
• Career development: Opportunities for professional growth and career progression.
• Collaborative culture: A supportive working environment with tools and practices that encourage teamwork, knowledge sharing, and continuous improvement.
• Inclusive workplace: A commitment to an inclusive environment where people with diverse backgrounds and perspectives are encouraged to apply. Reasonable accommodations can be arranged for candidates who require support during the recruitment process.