AI Engineer

Benchmarkit; Pune District; On-site; INR 600,000 - 1,200,000; Full time

Job summary

Benchmark IT Solutions India Pvt Ltd is hiring for a hands-on GenAI engineer. You will build end-to-end RAG pipelines, implement JSON extraction, and design REST APIs around LLM features. The role emphasizes practical engineering over research, with a focus on reliability and cost monitoring.

Ideal candidates bring 1–2 years of GenAI experience, strong Python skills, and familiarity with at least one LLM API and vector store. A collaborative team environment and ownership of features are offered.

Qualifications

• 1–2 years of hands-on experience building LLM or GenAI applications.
• Strong Python fundamentals; comfortable with FastAPI, Flask or Django.
• Experience with at least one LLM API and vector store.
• Understanding of RAG concepts and JSON schema validation.

Responsibilities

• Build RAG pipelines end to end from document ingestion to grounded answers with citations.
• Implement structured extraction from documents and semi-structured text into validated JSON.
• Design and consume REST APIs around LLM features and integrate with workflows.
• Implement tool calling and agent-like workflows with guardrails and human-in-the-loop steps.
• Write tests and evaluate retrieval quality and accuracy to prevent regressions.

Skills

Python

LLM development

API design

JSON/Pydantic

Tools

FastAPI

Flask

Django

OpenAI API

Azure OpenAI

Pinecone

Qdrant

Chroma

FAISS

pgvector

Job description

Benchmark IT Solutions India Pvt Ltd | Full time

Since our establishment in 2005, we have been focused on helping our clients build their Software Products/Projects.

Our founders had software development engineering background, our focus has always been in having strong delivery capability. This has resulted in extremely happy clients from day one and we continue to be the preferred partner for the clients we have acquired in our first year.

As we continued our journey and growth, we have expanded our team with different technology stacks and right technical solutions. Our 200+ people team continues to pleasantly surprise our clients across different technologies.

Job Description

The role

You will build GenAI-powered features that ship to real users: retrieval-augmented generation (RAG) services, structured data extraction from documents, LLM-backed APIs, and agent-style workflows with human review. You will also write the ordinary engineering that surrounds a model — validation, APIs, data access, tests and monitoring — because that is what makes an AI feature dependable. You will work in a team with a Technical Lead who reviews your designs and code.

This is a hands-on engineering role, not a research or prompt-writing role.

What you will do

Build RAG pipelines end to end: document ingestion, chunking, embedding, vector search, re-ranking, grounded answers with citations.

Implement structured extraction from documents and semi-structured text into validated JSON, with confidence handling and fallback.

Design and consume REST APIs around LLM features; integrate with existing applications and workflow pipelines.

Implement tool calling and Agentic workflows with clear guardrails and human-in-the-loop steps where needed.

Write evaluation sets and tests for LLM features — measure retrieval quality and answer accuracy, and catch regressions before release.

Track cost, latency and failures in production; debug why a request went wrong.

Follow good engineering practice: version control, code review, secrets kept out of code, readable and tested code.

Use AI coding assistants productively while remaining accountable for what you commit.

What you must bring

1–2 years of hands-on experience building LLM or GenAI applications, with at least one feature that real users have used — not only a notebook or a hackathon demo.

Strong Python fundamentals; comfortable with FastAPI, Flask or Django for building APIs.

Experience with at least one LLM API (OpenAI, Azure OpenAI, Anthropic, Gemini or open-weight models) and at least one vector store (pgvector, Qdrant, Pinecone, Chroma, Azure AI Search, MongoDB Atlas Vector Search, FAISS).

Working understanding of RAG concepts and failure modes: chunking trade-offs, retrieval misses, hallucination, prompt injection, context limits.

Structured outputs: function or tool calling, JSON schema, Pydantic validation.

Git, basic Docker, and the habit of writing tests.

Ability to explain, with reasons, when a rule or a simple lookup is a better choice than an LLM.

Good to have

LangChain, LangGraph, LlamaIndex, Semantic Kernel or a similar framework — and the judgment to use them sparingly.

Evaluation tooling (Ragas, DeepEval, promptfoo) or a harness you built yourself.

Cloud experience, Azure preferred; workflow orchestrators (Temporal, Airflow, Prefect).

Exposure to enterprise or regulated-industry projects; any of AI-102, AZ-204 or equivalent certifications.