Semantic Backend Engineer (Contract, Remote)

Infuse; Welkom; Remote; ZAR 500 000 - 700 000; Full time

Job summary

A leading tech company is looking for a candidate to manage the ETL pipeline for transforming raw PDFs into structured resources while utilizing advanced machine learning models. The ideal applicant has experience with semantic search, embeddings, and a passion for turning unstructured data into valuable insights. This role offers an opportunity to shape the relevance and freshness of a catalog used by over a million resources.

Qualifications

• Experience building ML pipelines impacting real users.
• Familiarity with semantic search and large-scale tagging.
• Ability to manage unstructured data effectively.

Responsibilities

• Own the ETL pipeline from raw PDFs to structured resources.
• Finalize summarization and classification flows using models.
• Implement filtering and mapping to topic taxonomies.
• Generate and query embeddings with specified tools.

Skills

Machine Learning pipelines

Semantic search

Embeddings

Python

PyTorch

FastAPI

Docker

Tools

OpenAI APIs

Milvus

pgvector

Airflow

Lambda

Athena

Redshift SQL

Job description

INFUSE is committed to complying with applicable data privacy and security laws and regulations. For more information, please see our Privacy Policy.

Overview

INKHUB is ingesting 10 million raw PDFs to build the Internet's richest catalog of marketing-grade B2B content — tagged, summarized, and searchable by topic, company, or intent.

What You’ll Do

• Own the ETL pipeline from raw PDFs (S3-ingested) to structured resources.
• Finalize our summarization and classification flow using open-source models with GPT-4o fallback.
• Apply filtering logic (e.g., 3‑year age, page count) to enforce resource quality.
• Map each asset to the specific topic taxonomy (10+ per topic across ~9 topics).
• Generate dense embeddings using sentence‑transformers.
• Load and query embeddings using Milvus or pgvector.
• Implement “freshness” logic to identify and index only new or updated content based on file diffing, crawl timestamp, or document hash.
• Build a QA/eval harness: format compliance, recall@5, drift monitoring.
• Expose /v1/semantic‑search via FastAPI, with filtering and rank fusion.
• Collaborate closely with our Tech Lead on UX integration and snippet generation.

Your Toolbox

Python, PyTorch, sentence‑transformers, OpenAI APIs, or similar pretrained LLMs. FastAPI, Milvus or pgvector, PyPDF/Tika, Airflow or Lambda for orchestration. Docker, GPU scheduling, Athena/Redshift SQL.

You Might Be a Fit If

• You’ve built ML pipelines that touched real users, not just notebooks.
• You’ve worked on semantic search, embeddings, or large‑scale tagging.
• You’ve wrestled with unstructured data and love turning chaos into clarity.
• You like working fast, iterating with feedback, and tracking metrics that matter.

Why This Role Matters

Your models decide what gets found, how it's tagged, and which content and companies stand out. You’ll help define what “relevance” and “freshness” mean for over a million resources and 50+ company pages and make sure INKHUB stays ahead of the curve.

Referrals increase your chances of interviewing at INFUSE by 2x.

Be among the first 25 applicants to get a fair and detailed assessment from our seasoned recruiting professionals.