Software Developer (Content Integration & Data Pipelines)

Jobgether; United States; Remote; USD 54,000 - 79,000; Full time

Benefits offered by this job

Remote work

Health coverage

RRSP matching

Life insurance

Employee assistance program

Paid time off

Holiday break

Meeting-free Fridays

Career development

Collaborative culture

Inclusive workplace

Job summary

Jobgether is seeking a data engineering-focused software engineer to build scalable web scrapers, content integration services, and Kafka-based pipelines. You will integrate external sources, develop Java-based backends, and ensure data quality in high-volume environments.

You will work remotely with a cross-functional team on architecture and performance optimizations, embracing innovation and advanced data processing techniques.

Qualifications

• Degree in Computer Science or related field, or equivalent practical experience.
• 3+ years of professional software development experience.
• Strong TypeScript or JavaScript skills for content integration, parsing, and data processing.
• Backend services development with Java or similar language.
• Production web scraping in production environments with responsible crawling practices.
• Experience with event-driven architectures, Kafka.
• Knowledge of HTML parsing, content extraction, data transformation, and normalization.
• High-volume, fault-tolerant data pipelines experience.
• SQL and data modeling knowledge.
• Linux, Git, Maven in modern workflows; Solr knowledge is a plus.
• Experience building content ingestion pipelines for news/media or data-intensive apps.
• Understanding distributed systems, scalability, reliability, and performance optimization.
• Ownership and initiative; can work independently.
• Analytical problem-solving and troubleshooting skills.
• Effective communication and teamwork.

Responsibilities

• Build scalable web scrapers and crawlers with responsible crawling.
• Develop content integration services using TypeScript/JavaScript.
• Design and maintain Kafka-based event-driven pipelines.
• Integrate external content sources, feeds, and APIs.
• Develop Java-based RESTful backend services.
• Improve data quality with validation and normalization practices.
• Monitor and optimize pipeline performance and costs.
• Contribute to system architecture and scalability decisions.
• Support search/indexing workflows with Solr integration.
• Collaborate with team and stakeholders; share knowledge.

Skills

TypeScript

JavaScript

Backend development

Kafka

Content processing

Data pipelines

SQL

Distributed systems

Problem solving

Communication

Teamwork

Innovation

Education

Bachelor’s degree in Computer Science or related field

Tools

Git

Maven

Linux

Solr

HTML parsing

Job description

Accountabilities

- Build scalable web scrapers and crawlers: Develop and maintain production-ready scraping and crawling solutions that collect online content responsibly, following standards such as robots.txt, rate limiting, and applicable terms of use.

- Develop content integration services: Implement parsing, extraction, and normalization logic using TypeScript and JavaScript to transform content from diverse sources into consistent, usable data.

- Design event-driven pipelines: Build and maintain Kafka-based services that process, enrich, and route content in near real time, supporting reliable and scalable data flows.

- Integrate external content sources: Connect third-party content providers, RSS and Atom feeds, and external APIs to expand and maintain content coverage.

- Develop backend services: Design and implement Java-based RESTful services that support content processing, integration, and delivery across the platform.

- Improve data quality and reliability: Establish and apply best practices for error handling, source management, validation, data normalization, and fault-tolerant pipeline operations.

- Monitor and optimize pipeline performance: Implement observability and monitoring practices, investigate processing issues, and improve system performance, reliability, and operating costs.

- Contribute to technical architecture: Participate in system design discussions, evaluate scalability trade-offs, and help define the technical direction of content acquisition and data processing systems.

- Support search and indexing capabilities: Contribute to content indexing and retrieval workflows, collaborating with search technologies such as Solr to ensure content is structured and accessible.

- Collaborate across the development team: Work with technical colleagues and other stakeholders to deliver maintainable solutions aligned with product and business needs.

- Mentor and share knowledge: Support junior developers, contribute to code reviews and engineering best practices, and promote continuous learning within the team.

Requirements:

- Education and experience: A degree in Computer Science or a related field, or equivalent practical experience, with at least three years of professional software development experience.

- TypeScript and JavaScript proficiency: Strong programming skills in TypeScript or JavaScript, particularly for content integration, parsing, and data processing.

- Backend development experience: Solid experience building backend services with Java or a comparable programming language.

- Production web scraping expertise: Hands-on experience developing and operating web scrapers and crawlers in production environments, with an understanding of robots.txt, rate limiting, responsible crawling practices, and applicable terms of use.

- Event-driven architecture: Practical experience with event-driven systems and message-streaming technologies, particularly Kafka.

- Content processing skills: Knowledge of HTML parsing, content extraction, data transformation, and normalization techniques.

- Data pipeline engineering: Experience designing or maintaining high-volume, fault-tolerant data pipelines that process and deliver information reliably.

- Database and data modeling knowledge: Familiarity with SQL and data modeling principles.

- Development tools and environments: Experience working in Linux environments and using Git and modern software development workflows. Familiarity with Maven is also expected.

- Search technology knowledge: Experience with Solr, including indexing, schema design, and query tuning, is an asset.

- Relevant industry experience: Experience building content ingestion pipelines for news, media, or similar data-intensive applications is strongly preferred.

- System design capabilities: Understanding of distributed systems, scalability trade-offs, reliability, and performance optimization.

- Ownership and initiative: Ability to work independently, take responsibility for projects, and proactively identify opportunities for technical improvement.

- Analytical problem-solving: Strong troubleshooting skills and the ability to investigate complex technical issues and develop practical solutions.

- Communication and teamwork: Ability to communicate effectively with technical and non-technical stakeholders, collaborate across teams, and share knowledge.

- Interest in innovation: Enthusiasm for data engineering, automation, emerging technologies, and AI-powered capabilities.

Benefits:

- Competitive compensation: Expected annual salary range of CAD $75,000–$110,000, depending on experience, skills, and internal equity.

- Remote work: Fully remote working environment.

- Health coverage: Health, dental, and vision benefits.

- Retirement savings: RRSP matching.

- Financial protection: Life insurance.

- Employee support: Access to an Employee Assistance Program (EAP).

- Paid time off: Paid vacation, personal days, and sick days.

- Holiday break: A week off between Christmas and New Year's.

- Meeting-free Fridays: No internal meetings scheduled on Fridays.

- Career development: Opportunities for professional growth and career progression.

- Collaborative culture: A supportive working environment with tools and practices that encourage teamwork, knowledge sharing, and continuous improvement.

- Inclusive workplace: A commitment to an inclusive environment where people with diverse backgrounds and perspectives are encouraged to apply. Reasonable accommodations can be arranged for candidates who require support during the recruitment process.