Machine Learning Engineer / SME (LLM Eval & RAG Testing) – SME | TS/SCI

Falls Technology; McLean (VA); On-site; USD 120,000 - 200,000; Full time

Benefits offered by this job

Retirement plan (9% employer match)

Medical premiums paid for family

4 weeks PTO

Job summary

Falls Technology is seeking a results-driven AI/ML engineer to prove which LLMs work for mission-focused applications. On-site in McLean, VA, you will design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships trusted AI systems.

You will test, evaluate, and validate AI systems, develop benchmarks, and communicate results to both technical and non-technical stakeholders, leveraging Python/R, PyTorch, TensorFlow, and CI pipelines in

Qualifications

• Active TS/SCI clearance is required.
• Hands-on RAG testing experience.
• Tuning neural nets/LLMs on custom datasets for real use cases.
• Building AI/ML models and ensembles in Python or R.
• Proficiency with PyTorch, TensorFlow or Keras.
• GPU-accelerated computing proficiency.
• Testing, evaluating or validating LLM-based AI systems.
• Experiment design to benchmark LLMs.
• SQL including CTEs, set operations, aggregates and nested subqueries.
• GitHub and Jenkins CI for reproducible experiments.

Responsibilities

• Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing).
• Design and run experiments that benchmark LLMs against defined measures.
• Tune neural networks / LLMs on custom datasets and apply results to concrete use cases.
• Build AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment).
• Use deep learning frameworks such as PyTorch, TensorFlow or Keras.
• Leverage GPUs for accelerated training and evaluation.
• Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets.
• Work in GitHub and Jenkins-style CI for reproducible experiments.
• Communicate methodological choices and model results to technical and non-technical stakeholders.

Skills

RAG testing

LLM evaluation

Python/R

PyTorch/TensorFlow/Keras

GPU acceleration

SQL (CTEs / subqueries)

GitHub / CI

Experiment design

Tools

TensorFlow

Keras

Hugging Face Transformers

Job description

Active TS/SCI required. U.S. citizenship required.

Location: On-site in McLean, VA (Full-Time).

Prove which LLMs actually work for the mission. You'll design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships AI systems they can trust.

What you'll do

• Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing)


• Design and run experiments that benchmark LLMs against defined measures


• Tune neural networks / LLMs on custom datasets and apply results to concrete use cases


• Build AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment)


• Use deep learning frameworks such as PyTorch, TensorFlow or Keras


• Leverage GPUs for accelerated training and evaluation


• Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets


• Work in GitHub and Jenkins-style CI for reproducible experiments


• Communicate methodological choices and model results to technical and non-technical stakeholders

What you bring (required)

• Active TS/SCI


• Hands-on RAG testing experience (must-have)


• Tuning neural nets / LLMs on custom datasets for real use cases


• Building AI/ML models and ensembles in Python or R


• PyTorch, TensorFlow or Keras (key)


• GPU-accelerated computing (key)


• Testing, evaluating or validating LLM-based AI systems (key)


• Experiment design to benchmark LLMs (key)


• SQL including CTEs, set ops, aggregates and nested subqueries


• GitHub and Jenkins (or equivalent) version control / CI

Nice to have

• Hugging Face Transformers library and hub


• Building benchmarks that measure LLM performance


• Cloud computing development and architecture


• Flask or similar front-end/web frameworks


• Text classification and topic modeling (scikit-learn or deep learning)


• Semantic search applications


• Tableau dashboards


• Verification/validation test benches, Explainable AI (XAI), ONNX

Why Falls

A 9% retirement contribution, 100% of family medical premiums paid, and 4 weeks of PTO. We're a small business that's been delivering since 2019, so your work is visible and your voice counts.

Falls Technology is an Equal Opportunity Employer. All qualified applicants will receive consideration without regard to race, color, religion, sex, national origin, disability, or protected veteran status.