Senior Data Engineer - Cloud Ingestion & AI Knowledge Bases

Solvd; United States; Remote; USD 130,000 - 180,000; Full time

Job summary

Solvd Inc. seeks a Senior Data Engineer to own the ingestion pipeline across four source systems, turning raw knowledge content into a governed, permission-tagged corpus in S3 and Bedrock Knowledge Bases.

You’ll work end-to-end within an architecture set by the Solution Architect, alongside a retrieval engineer to ensure no delivery is single-person dependent. The role focuses on building connectors (Oracle B2C Service, Graph-driven SharePoint/OneDrive/Teams, and Sitecore), implementing the

Qualifications

• 4+ years building data pipelines in production, at least 2 on AWS with Python.
• Hands-on with Microsoft Graph API for SharePoint and Teams content — delta queries, permissions, and throttling.
• Experience landing unstructured content (HTML, Office documents, PDF, chat transcripts) into S3 and preparing for search or vector indexing.

Responsibilities

• Build the iKNOW connector against Oracle B2C Service knowledge APIs with full inventory extract and incremental sync.
• Build SharePoint, OneDrive, and Teams connectors on Microsoft Graph with delta queries and permission metadata capture.
• Build the Sitecore connector against its content APIs for public web content in scope.
• Implement the ingestion pipeline — landing in S3 with per-source prefix convention and chunking parameters.
• Produce inventory extracts for disposition reviews and counts for Phase-1 re-baselining.
• Implement retirement handling to remove retired/deleted items on next sync.
• Handle throttling/backoff for Oracle and Microsoft APIs; ensure idempotent re-runs and fault isolation.
• Instrument all pipeline stages with CloudWatch metrics and write runbooks.
• Pair with client IT during weeks 7–10 to enable future extensions.

Skills

Python

AWS

Microsoft Graph API

Data pipelines

Tools

Terraform

AWS CDK

CloudWatch

Job description

Solvd Inc. seeks a Senior Data Engineer to own the ingestion pipeline across four source systems, turning raw knowledge content into a governed, permission-tagged corpus in S3 and Bedrock Knowledge Bases.

You’ll work end-to-end within an architecture set by the Solution Architect, alongside a retrieval engineer to ensure no delivery is single-person dependent. The role focuses on building connectors (Oracle B2C Service, Graph-driven SharePoint/OneDrive/Teams, and Sitecore), implementing the