Head of Data Pipelines & Web Crawling, Content AI

Jobgether; United States; Remote; USD 104,000 - 118,000; Full time

Benefits offered by this job

Fully remote work environment

Health, dental, and vision insurance

RRSP matching

Life insurance

Employee Assistance Program (EAP)

Career development and progression

Paid vacation, personal days, and sick

Week off between Christmas and NewYear

No internal meetings on Fridays

Job summary

Jobgether is seeking a senior engineering leader to guide a team of developers in building and scaling web crawling, data ingestion, and mention-matching systems. You will define technical strategy, collaborate with product stakeholders, and drive reliable data pipelines while maintaining high quality and security standards.

You will lead architecture discussions, oversee incident response, and partner with Customer Success to resolve client gaps in media coverage.

Qualifications

• 7+ years of software development including web crawling, data ingestion, or large-scale data pipelines.
• 2+ years of experience managing software development teams.
• Bachelor's degree in Computer Science or related discipline, or equivalent practical experience.
• Strong technical expertise in Java or a comparable type-safe programming language.
• Experience designing distributed systems, including message queues or streaming technologies such as Kafka.
• Familiarity with cloud platforms such as AWS, GCP, or Azure, containerization technologies like Docker and Kubernetes, and modern CI/CD practices.
• Knowledge of search technologies such as Elasticsearch or Solr, along with Boolean query logic.
• Experience integrating social media platform APIs and working with third-party data providers.
• A solid understanding of legal, ethical, and privacy considerations associated with web and social data collection.
• Experience working in Agile development environments and using tools such as JIRA and Confluence.
• Strong communication and collaboration skills, with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
• A data-driven approach to leadership, using metrics to guide technical decisions, assess performance, and measure team outcomes.
• A supportive leadership style that balances delivery expectations with team development, engagement, and well-being.
• Experience with NLP, large language models (LLMs), entity recognition, or sentiment analysis is an asset.
• Experience in PR technology, media monitoring, social listening, or the newswire industry is an advantage.

Responsibilities

• Lead, mentor, and develop a team of software developers, establishing objectives and fostering accountability and excellence.
• Define the technical strategy and architecture for web crawling, data ingestion, and mention-matching systems.
• Collaborate with Product Owners to translate needs into a prioritized technical roadmap.
• Ensure scalable, reliable, and cost-effective data pipelines as content grows.
• Establish and monitor performance metrics across coverage, accuracy, latency, uptime, and quality.
• Oversee incident response and on-call practices; drive root-cause analysis and improvements.
• Participate in architecture discussions, reviews, and hands-on troubleshooting.
• Partner with Customer Success to address client gaps in media coverage.
• Ensure data collection complies with crawling directives, terms of service, copyright, licensing, and privacy laws; consult Legal when needed.
• Assess emerging technologies and data providers to improve coverage and capabilities.
• Manage relationships with third-party data vendors and API partners with Content Partnerships.
• Support evolution of AI-enabled workflows using comprehensive content data for insights and alerts.

Skills

Leadership

Team management

Java

Distributed systems

Cloud platforms

CI/CD

Elasticsearch/Solr

NLP/LLMs

Education

Bachelor's degree in CS or related

Tools

Kafka

Docker

Kubernetes

Elasticsearch

Solr

AWS

GCP

Azure

Job description

Jobgether is seeking a senior engineering leader to guide a team of developers in building and scaling web crawling, data ingestion, and mention-matching systems. You will define technical strategy, collaborate with product stakeholders, and drive reliable data pipelines while maintaining high quality and security standards.

You will lead architecture discussions, oversee incident response, and partner with Customer Success to resolve client gaps in media coverage.