AI Inference Engineer

ITRex Group; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Remote flexibility

Medical benefits

Learning budget

Job summary

ITRex Group is seeking a hands-on C++ engineer to develop and optimize the on-device AI stack. You will port and enhance inference engines such as llama.cpp or ggml to run efficiently on edge devices and across different hardware, focusing on runtime, model loading, and performance.

This role suits engineers who enjoy low-level work and private, fast AI without cloud dependence. You’ll collaborate with researchers to translate research into production and enrich products with the latest ML

Qualifications

• Proficiency in C++ with strong systems programming background.
• Experience with AI/ML concepts, transformers, and LLMs is preferred.
• Familiarity with edge devices and on-device inference frameworks.

Responsibilities

• Deploy ML models to edge devices using llama.cpp and ggml frameworks.
• Collaborate with researchers to move models from research to production environments.
• Integrate AI features into existing products and optimize runtime performance.
• Ensure stability and performance of the inference layer across hardware platforms.
• Demonstrate ability to assimilate new technologies quickly.

Skills

C++

JavaScript

Llama.cpp

ggml

Deep learning

Transformers/LLMs

Edge AI

Education

B.Sc. in Computer Science/AI/ML

Tools

llama.cpp

ggml

Job description

About ITRex

THE PLACE

ITRex - AI pioneers who build systems that actually work in the real world, not just in demos. We’re 250+ people spread across the US and Europe, creating solutions for companies like Procter & Gamble and Shutterstock. We keep it simple, build it right, and focus on what works.

THE PEOPLE

We’re the kind of people who don’t ignore messages in Slack, who jump in to help when you’re stuck on a problem, and who offer solutions instead of blame when things go sideways. We believe in openness, accountability, and having each other’s backs. No office politics, no hidden agendas - just people who care about doing good work together and supporting each other to get there.

THE ROLE

You’ll work on the C++ layer that powers local AI, porting and enhancing inference engines like llama.cpp or similar, to run efficiently on edge devices. Your focus is on the runtime: making models load faster, run leaner, and perform well across different hardware. You’ll ensure that the inference layer is stable, optimized, and ready for integration with the rest of the stack.

This role is for engineers who want to work close to the metal, enabling private and fast on-device AI without relying on cloud infrastructure.

Responsibilities

• Work on deploying machine learning models to edge devices using the frameworks: llama.cpp, ggml

• Collaborate closely with researchers to assist in coding, training and transitioning models from research to production environments

• Integrate AI features into existing products, enriching them with the latest advancements in machine learning

• Excellent programming skills in C++, experience in Javascript is a bonus

• Strong experience with Llama.cpp and ggml inference engines, which facilitates the deployment of models to specific GPU architectures

• Good understanding of deep learning concepts and model architectures

• Experience with transformers, LLMs, Diffusion models

• Demonstrated ability to rapidly assimilate new technologies and techniques

• A degree in Computer Science, AI, Machine Learning, or a related field, complemented by a solid track record in AI R&D

Why people stay

First, the foundation:

• Remote flexibility: Work where and how you work best - we trust you to deliver
• Fair compensation: Competitive salary + benefits that matter (medical, learning)

Then, the growth:

• Ownership opportunities:See a problem worth solving? Own it. We back smart risks over bureaucratic safety
• AI enhancement: We leverage AI to make you faster and stronger - complementing your abilities, not replacing them
• Learning investment: English classes, professional development
• Career progression: Real paths up, not just sideways shuffling

Finally, the people:

• Responsive teammates: No ignored Slacks, no “not my problem” attitudes
• Supportive culture: When you’re stuck, people help. When things break, we fix them together
• Human connections: Regular meetups, tech talks, and actual relationships beyond work

Curious? We are too. Let’s talk