Senior Data Engineer (Agentic Retrieval & Memory)

Coforge; Oaks (PA); On-site; USD 130,000 - 190,000; Full time

Job summary

Coforge is seeking a Senior Data Engineer (Agentic Retrieval & Memory) in Oaks, PA for a full-time role. You will design agent memory interfaces across session state, short-term history, and long-term memory, and select optimal stores for each tier.

You will integrate with the enterprise search platform and evaluate backing stores for performance and cost. You will leverage 5+ years in data engineering, 3+ years in agent memory patterns, and 3+ years with vector stores to build scalable,

Qualifications

• 5+ years hands-on data engineering in production environments.
• 3+ years working on agentic or LLM data patterns, with depth in agent memory.
• 3+ years with vector and semantic search using modern vector stores.
• 3+ years designing NoSQL, document, or key-value stores for high-write workloads.
• 2+ years with caching or in-memory stores like Redis.
• 3+ years Python to production standard.
• 2+ years on a major cloud platform.

Responsibilities

• Design and build the agent memory interface across tiers: session state, short-term history, and long-term memory.
• Select and implement appropriate stores for each memory tier (cache, document store, vector store).
• Design and build retrieval interface over the enterprise search platform via shared SDK.
• Assess current backing stores against workload, including partitioning, TTL, retention, read/write patterns, latency, and cost.

Skills

Data engineering

Agent memory patterns

Vector search

NoSQL design

Caching stores

Python production

Cloud platforms

RAG retrieval

Tools

Azure AI Search

PostgreSQL with pgvector

Elasticsearch

Redis

Job description

Role: Senior Data Engineer (Agentic Retrieval & Memory)

Mode of Hire: Full time

RESPONSIBILITIES

• Design and build the agent memory interface across the tiers an agent actually needs: working or session state within a run, short-term conversation history, and long-term memory that persists across sessions. This includes what is written, what is summarized or compacted, what expires, and how state is isolated between users and threads.
• Select and implement the right store for each memory tier: a cache or in-memory store for volatile session state, a document or key-value store for conversation history, and a vector store for semantic long-term recall. Match the store to the access pattern rather than forcing one store to serve every tier.
• Design and build the retrieval interface over the client’s existing enterprise search platform, exposed through the shared SDK so agents query it consistently rather than wiring their own integrations.
• Assess the current backing stores against the workload: partitioning strategy, item and document size constraints, time-to-live and retention, read and write patterns under conversational load, latency inside a live agent loop, and cost at volume.

Skills Required

• 5+ years hands‑on data engineering in production environments, covering data modelling, storage design, query performance and the operational behaviour of the stores you choose.
• 3+ years working on agentic or LLM data patterns, with real depth in agent memory: session and conversation state, short‑term and long‑term memory, summarisation and compaction, expiry and retention, and isolation between users and threads. This is the defining requirement: conventional data engineering alone is not sufficient for this position.
• 3+ years with vector and semantic search, using Azure AI Search, PostgreSQL with pgvector, Elasticsearch, or a comparable vector store, including hybrid search, relevance tuning and index design.
• 3+ years designing NoSQL, document or key‑value stores for high‑write, low‑latency workloads: partitioning and sharing strategy, item and document size constraints, time‑to‑live and retention, and the read and write patterns of conversational or session‑based data.
• 2+ years working with caching or in‑memory stores such as Redis or equivalent, for volatile and episodic state, including expiry strategy and the trade‑offs against durable storage.
• Working knowledge of retrieval‑augmented generation, including chunking and embedding strategy and how model choice and chunking affect retrieval quality and cost. You will consume an existing vectorised knowledge base more often than you build one.
• 3+ years Python to production standard, building interfaces or libraries consumed by other engineers rather than scripts.
• 2+ years working on a major cloud platform, including managed data services, identity and access to data stores, and private networking to data services.
• Experience evaluating retrieval and memory quality, using groundedness, relevance or comparable measures, rather than relying on subjective assessment.

Preferred skillset

• Experience with managed agent memory services or memory frameworks such as those offered by agent platforms, and a view on when to use them rather than building directly on a store.
• Experience with agent frameworks and how retrieval and memory are consumed inside an agent loop.
• Knowledge graph or entity resolution approaches to long‑term or structured memory.
• Experience in financial services or another regulated industry, including data residency, retention and the handling of sensitive data.
• Familiarity with OpenTelemetry or platform observability tooling, particularly tracing retrieval and memory calls inside agent runs.
• Exposure to Model Context Protocol (MCP) or comparable patterns for exposing data sources to agents.
• Experience with data catalogues or lineage tooling in an enterprise setting.