Forward Deployed Engineer

Josh Talks Pvt Ltd.

Gurugram District

On-site

INR 1,200,000 - 1,800,000

Full time

13 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Josh Talks Pvt Ltd. is seeking an Forward Deployed AI/ML Engineer — Evaluations (Product + Engineering) to own how we measure the quality of speech, voice, and image data, and to drive evaluation plans end to end.

You will work with ASR, TTS, S2S, and voice-agent models, running experiments on cloud GPUs and coordinating with research, product, and business teams. You will design evaluation plans, set up tests, and produce actionable insights with clear reports and dashboards.

Qualifications

  • Hands-on evaluation of ASR, TTS, S2S, or voice/LLM agent models.
  • Strong Python with Pandas/NumPy, Hugging Face, and audio libraries (librosa, torchaudio).
  • Running ML model inference (PyTorch / Hugging Face) on GPUs.
  • Working with third-party model APIs (REST, WebSocket/streaming).
  • Cloud platforms (GCP, AWS, or Azure) with GPU instances.
  • Evaluation metrics, test-set design, and statistical significance.
  • Designing human evaluation studies and rating guidelines.
  • Clear written and verbal communication, and explaining results to technical and non-technical people.
  • Strong ownership and ability to work independently in a fast-moving team.

Responsibilities

  • Own evaluation roadmap across ASR, TTS, S2S, and voice-agent models.
  • Define quality meanings per use case and translate into evaluation plans.
  • Run end-to-end evaluations and manage inference on cloud GPUs.
  • Integrate models from different vendors via APIs and compare fairly.
  • Build evaluation datasets including multilingual and noisy audio.
  • Design and execute human evaluation studies and guidelines.
  • Publish results via reports and dashboards; advise go/no-go decisions on releases.
  • Identify error patterns and feed insights back to teams.

Skills

ASR/TTS/S2S evaluation
Python
Pandas/NumPy
Hugging Face
Audio libraries (librosa/torchaudio)
PyTorch
Cloud platforms (GCP/AWS/Azure)
Model inference on GPUs
APIs (REST/WS)
Evaluation metrics & test-set design
Human evaluation design
Communication skills
Ownership & independence

Tools

GPU instances

Job description

Forward Deployed AI / ML Engineer — Evaluations (Product + Engineering)

Website: ai.joshtalks.com and voicearena.com

Location: Gurugram

About us

Josh Talks AI builds and licenses audio datasets to leading AI labs. We’ve been deliberately quiet for the last year and just built. The product has earned the right to be expanded; we are now hiring the people who will own what gets built next.

About the role

This is an Evaluations PM + Engineer role. You will own how we measure the quality of our speech, voice, and image data, and of our customers' models: ASR, TTS, speech-to-speech (S2S), and voice agents. You will define what "good" means for each dataset and model, build and run the evaluations yourself, and turn the results into clear decisions for the product, research, and business teams.

You don't hand evaluations off. You plan them, run them end to end, and own the outcome.

What you'll do
Evaluation product ownership
  • Own the evaluation roadmap across ASR, TTS, S2S, and voice-agent models.
  • Work with research, product, and business teams to understand use cases and define what quality means for each one.
  • Turn quality goals into clear evaluation plans: test sets, metrics, pass/fail thresholds, and timelines.
  • Prioritise which models, languages, and use cases to evaluate first, based on impact.
  • Share results through clear reports and dashboards, and make go/no-go recommendations on model releases.
  • Track model quality over time and flag regressions early.
Hands-on evaluation (done by you)
  • Run end-to-end evaluations of ASR, TTS, S2S, and voice-agent models yourself.
  • Set up and run open-source and in-house models locally and on cloud GPUs for inference.
  • Integrate and run models from different vendors and customers through their APIs (batch and streaming), and compare them fairly.
  • Manage cloud GPU instances (GCP/AWS/Azure) to run large-scale evaluation and inference jobs efficiently.
  • Build and curate evaluation datasets, including multilingual, Indian-language, noisy, and real-world audio.
  • Design and run human evaluations: listening tests, rating guidelines, annotator training, and agreement checks.
  • Write reproducible evaluation pipelines in Python and automate recurring benchmarks.
  • Analyse errors to find failure patterns and feed them back to the research and data teams.
  • Benchmark in-house models against third-party and open-source models.
What we're looking for
Skills
  • Hands-on evaluation of ASR, TTS, S2S, or voice/LLM agent models.
  • Strong Python, with Pandas/NumPy, Hugging Face, and audio libraries (librosa, torchaudio).
  • Running ML model inference (PyTorch / Hugging Face) on GPUs.
  • Working with third-party model APIs (REST, WebSocket/streaming).
  • Cloud platforms (GCP, AWS, or Azure), including GPU instances.
  • Evaluation metrics, test-set design, and statistical significance.
  • Designing human evaluation studies and rating guidelines.
  • Clear written and verbal communication, and explaining results to technical and non-technical people.
  • Strong ownership, and the ability to work independently in a fast-moving team.
Nice to have
  • Experience fine-tuning speech models (ASR, TTS, or S2S).
  • Experience with Indian languages, code-mixed, or low-resource speech data.
  • Familiarity with Whisper, NeMo, pyannote, ESPnet, or commercial speech APIs.
  • Experience building evaluation dashboards or automated regression testing.
  • Own how quality is defined and measured for speech AI used at scale.
  • Work closely with research and product, and directly shape which models ship.
  • High ownership and real impact from day one.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Applied AI Engineer
Applied AI Engineer

Synth (YC S21) • Bengaluru

On-site
INR 1,500,000 - 2,500,000
ML Research Engineer, Speech
ML Research Engineer, Speech

Blue Machines AI • Bengaluru

On-site
INR 1,200,000 - 2,000,000
ML Research Engineer, Speech
ML Research Engineer, Speech

Blue Machines AI • Bengaluru Urban

Hybrid
INR 900,000 - 1,300,000
Senior ML Research Scientist, Speech
Senior ML Research Scientist, Speech

Blue Machines AI • Bengaluru Urban

Hybrid
INR 3,000,000 - 6,000,000
Evals Engineer
Evals Engineer

PingAura AI Technologies Private Limited. • Mumbai

On-site
INR 1,000,000 - 1,500,000
Founding-tier ownership
Direct collaboration with AI experts
Competitive salary
ML Engineer - Speech
ML Engineer - Speech

Rezo.AI • Dadri

On-site
INR 1,200,000 - 2,400,000
Senior ML Research Scientist, Speech
Senior ML Research Scientist, Speech

Blue Machines AI • Bengaluru

Hybrid
INR 3,500,000 - 6,000,000
Senior AI Engineer - Agentic AI and Voice Agents
Senior AI Engineer - Agentic AI and Voice Agents

YAL • Hyderabad

On-site
INR 3,500,000 - 5,500,000
Solutions Architect – Speech AI
Solutions Architect – Speech AI

Gnani Inc. • Bengaluru

On-site
INR 4,200,000 - 6,800,000
Tech Lead — ASR / TTS / Speech LLM (IC + Mentor)
Tech Lead — ASR / TTS / Speech LLM (IC + Mentor)

OutcomesAI • Bengaluru

On-site
INR 4,000,000 - 7,000,000