LLM Evaluation & RAG Benchmark Engineer (TS/SCI)

Fallstechnology; McLean (VA); On-site; USD 120,000 - 200,000; Full time

Benefits offered by this job

Retirement plan (9% employer match)

Medical premiums paid for family

4 weeks PTO

Job summary

Falls Technology is seeking a results-driven AI/ML engineer to prove which LLMs work for mission-focused applications. On-site in McLean, VA, you will design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships trusted AI systems.

You will test, evaluate, and validate AI systems, develop benchmarks, and communicate results to both technical and non-technical stakeholders, leveraging Python/R, PyTorch, TensorFlow, and CI pipelines in

Qualifications

• Active TS/SCI clearance is required.
• Hands-on RAG testing experience.
• Tuning neural nets/LLMs on custom datasets for real use cases.
• Building AI/ML models and ensembles in Python or R.
• Proficiency with PyTorch, TensorFlow or Keras.
• GPU-accelerated computing proficiency.
• Testing, evaluating or validating LLM-based AI systems.
• Experiment design to benchmark LLMs.
• SQL including CTEs, set operations, aggregates and nested subqueries.
• GitHub and Jenkins CI for reproducible experiments.

Responsibilities

• Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing).
• Design and run experiments that benchmark LLMs against defined measures.
• Tune neural networks / LLMs on custom datasets and apply results to concrete use cases.
• Build AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment).
• Use deep learning frameworks such as PyTorch, TensorFlow or Keras.
• Leverage GPUs for accelerated training and evaluation.
• Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets.
• Work in GitHub and Jenkins-style CI for reproducible experiments.
• Communicate methodological choices and model results to technical and non-technical stakeholders.

Skills

RAG testing

LLM evaluation

Python/R

PyTorch/TensorFlow/Keras

GPU acceleration

SQL (CTEs / subqueries)

GitHub / CI

Experiment design

Tools

TensorFlow

Keras

Hugging Face Transformers

Job description

Falls Technology is seeking a results-driven AI/ML engineer to prove which LLMs work for mission-focused applications. On-site in McLean, VA, you will design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships trusted AI systems.

You will test, evaluate, and validate AI systems, develop benchmarks, and communicate results to both technical and non-technical stakeholders, leveraging Python/R, PyTorch, TensorFlow, and CI pipelines in