Skip to main content

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

GenAI: Code Challenge (Pilot)

Evaluating AI-generated test code.

Overview

The NIST GenAI Pilot Code Challenge will measure and evaluate unit tests generated by Artificial Intelligence (AI) for testing elementary-level python code. This pilot provides an evaluation framework that facilitates the development and improvement of AI Large Language Models (LLMs) technologies to write effective tests for software code. The Pilot Code Challenge is open to all who wish to participate and abide by the rules and procedures described throughout this website and evaluation plan.


What

The Code task is to automatically generate high-quality tests for python code given a textual specification of the task the python code is intended to carry out. There will be two prompt conditions: fixed prompt and custom prompt. Participants will be given a fixed prompt for each trial_id, this fixed prompt will already have the textual specification incorporated. The submission must contain an output that used the fixed prompt. Participants will have the opportunity to customize the prompts, and will be allowed to submit up to 9 alternative custom prompts in a submission. Both the fixed prompt and 1 custom prompt per trial_id are required in a submission. These text specifications may include a method header with all the parameter names and a brief summary of what the code inputs and outputs are and the function of the code. For more details, please see the GenAI Code Pilot Evaluation Plan.


Who

Teams from academia, industry, and other research labs are invited and encouraged to contribute to Generative AI research through the GenAI platform. This platform is designed to support various modalities, including both "Generators" and "Discriminators" technologies. The GenAI Code specifically focuses on Generators and measuring the quality of AI-generated test code.


How

To take part in the GenAI Code evaluation, participants must register on this website and agree to the license to download data and upload submissions. NIST will make all necessary data resources available. Please refer to the published schedule for data release dates. Participants will be able to upload their AI-generated test code to the challenge website and see their results displayed on the scoreboard.


Task Coordinator

If you have any questions, please email to the NIST GenAI team

Schedule

Date

Milestone

July 16, 2025

Evaluation Plan Posted

July 23, 2025

Registration and Submissions Open

September 12, 2025

Registration and Submissions Close

September 26, 2025

GenAI Code Preliminary Results Available (Leaderboard)

TBD

GenAI Code Pilot Workshop

Code Generation Instructions

GenAI Code Participants are provided the input json file containing the code bank input specifications, separated and labelled by trial_id, and the corresponding fixed prompts. With this input file, GenAI Code Participants will submit a json file of test outputs containing pytest tests that can test arbitrary implementations of the methods specified by the input specifications. This json file will have outputs separated and labelled by trial_id and by prompt_number. All tests generated using the fixed prompts will have prompt_number 0. Please see the GenAI Code Pilot Evaluation Plan for detailed instructions.

Only GenAI Code participants who have completed and submitted all required forms will be allowed access. Please check the published schedule for testing data release dates.

Optional Dry-Run Track

In addition to the main pilot track, there is a dry-run track on a small development data set. For this development data set, participants will be given not just the input problem file but also the key file, one baseline submission, and correct and incorrect code implementations. Participants may submit to the dry-run track and those submissions will be scored; however, the dry-run submissions will neither be analyzed nor scored on a scoreboard. All submissions to the dry-run track are optional.

Example Fixed Prompt


"We have python code that implements the following specification.

Specification:

def add(x: int, y: int) -> int:
    """
    Given two integers x, and y, return the sum of x and y. If either x or y is not
    an integer, raise a TypeError Exception.
    """


Please write python pytest test code that comprehensively tests the code for method add to determine if the code correctly meets the specification or not. When writing tests:
* write a comprehensive test suite,
* test edge cases,
* only generate tests that you are confident are correct, and
* include tests for TypeError cases.

Please write '###|=-=-=beginning of tests=-=-=|' before the tests. Write '###|=-=-=end of tests=-=-=|' immediately after the tests. Import any needed packages, including pytest. Additionally import the code being tested by adding the line `from genai_code_file import add` the line after '###|=-=-=beginning of tests=-=-=|'. Do not provide an implementation of the method add with the tests."
       

Submission Guidelines

  • Please refer to the GenAI Code Pilot Evaluation Plan for specific instructions. Please see the dry-run submission provided with the data for an example submission format.
  • Code should be compilable so that it can be run with the pytest package.
  • Each submission should be submitted as a single .json file, and each file should contain both the fixed prompt and custom prompt submissions.
  • For all submissions and all problems, the prompt used must be provided as metadata within the submission.
  • Submission notes: according to the published schedule, the submission page (form) will be open and available (via the GenAI website) for teams to submit their data outputs. Please make sure to follow the schedule and submit on time as extending the submission dates may not be possible.
  • Upon submission, NIST will validate the data outputs uploaded and report any errors to the submitter.
  • Please take into consideration that submitting your data outputs indicates and assumes your agreement to the "Protocol and Rules" section of the Evaluation Plan.

Example Sample Output:


from genai_code_file import add
import pytest

class TestCode(object):
    def test_add(self):
        assert add(2, 3) == 5
        with pytest.raises(TypeError):
            add("abc", "def")
      

Submission Validation

  • NIST will provide a validator script to participants to validate their output json files format as well as content specific to the task guidelines (e.g. required attributes). The submission platform will run this validator and fail any submissions that do not pass this validator.
  • Provided with the validator scripts are the scoring scripts (to allow participants to locally score dry-run submissions) and optional utility scripts that may help with certain components.

For the best user experience with the scoreboard, please use Google Chrome.
For the columns: finds CI1 error (%), finds CIT error (%), finds CI1 & CIT errors (%), and 100% coverage & finds all errors (%), only programs with correct tests were counted. For mean coverage (%), only programs with correct test were measured for coverage.

Coming Soon.