LLM Evaluation Expert (Coding)

Lancesoft; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

A leading company in AI innovation is looking for an LLM Evaluation Expert – Coding to evaluate code generated by AI models. This role involves analyzing code quality, providing feedback, and collaborating with researchers to improve coding standards. Candidates should have at least a bachelor’s degree and experience in software development across multiple programming languages.

Qualifications

• 1+ years of professional experience in software development.
• Strong understanding of software engineering principles and coding standards.
• Familiarity with AI/ML concepts related to LLMs and code generation.

Responsibilities

• Analyze and evaluate AI-generated code across multiple languages.
• Select optimal code solutions based on performance and correctness.
• Collaborate with AI researchers to enhance coding capabilities.

Skills

Software Development

Technical Evaluation

Code Review

Technical Writing

Education

Bachelor’s or Master’s degree in Computer Science

Tools

Python

Java

JavaScript

C++

Job description

Job Title: LLM Evaluation Expert Coding

Location: Remote

Company Overview:

AGI Data Services is at the forefront of AI innovation, specializing in the development and refinement of large language models (LLMs). Our mission is to build intelligent systems capable of understanding and generating high-quality code, revolutionizing the software development process.

Job Summary:

We are seeking a highly skilled LLM Evaluation Expert – Coding to assess and improve the performance of AI-generated code across multiple programming languages and paradigms. You will play a critical role in evaluating the technical accuracy, efficiency, and quality of code produced by LLMs. Your insights will directly influence the evolution of our next-generation AI coding assistants.

Key Responsibilities:

- Analyze and evaluate code generated by LLMs across different programming languages and styles.

- Select the best solutions from multiple AI-generated code options based on correctness, performance, and best practices.

- Write high-quality code demonstrations that set benchmarks for excellence in AI-generated programming output.

- Provide constructive, detailed feedback on code quality to enhance model performance.

- Collaborate with AI researchers to identify patterns, gaps, and improvement areas in the model's coding capabilities.

- Contribute to the development of coding standards, style guides, and evaluation frameworks.

- Stay updated with emerging trends in software engineering, machine learning, and AI-assisted development tools.

Required Qualifications:

- Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.

- 1+ years of professional experience in software development across multiple languages (e.g., Python, Java, JavaScript, C++, etc.).

- Strong understanding of software engineering principles, coding standards, and design patterns.

- Experience in technical evaluation, code review, or setting quality benchmarks for development.

- Familiarity with AI/ML concepts, especially related to LLMs, NLP, and code generation.

- Strong technical writing and communication skills, with a proven ability to explain code clearly and concisely.

Preferred Qualifications:

- Prior experience working with or evaluating AI/LLM-based code generation systems.

- Experience in developing or maintaining internal coding guidelines or documentation.

- Exposure to open-source contributions or writing developer tutorials.

- Understanding of machine learning workflows or data science platforms.

If you have any queries, you can reach out to me at any time.