2025 GenAI Code

Overview

The NIST GenAI Pilot Code Challenge will measure and evaluate unit tests generated by Artificial Intelligence (AI) for testing elementary-level python code. This pilot provides an evaluation framework that facilitates the development and improvement of AI Large Language Models (LLMs) technologies to write effective tests for software code. The Pilot Code Challenge is open to all who wish to participate and abide by the rules and procedures described throughout this website and evaluation plan.

This GenAI Code Pilot will function in two rounds, where the first round is called Round-1 and the second round is called Round-2. Unless stated otherwise, all instructions are the same for both rounds. Round-2 will follow the same protocols and procedures as the initial Round-1 pilot, and the Round-2 input and output data formats will be the same as used in Round-1. Although Round-2 will require new submissions as the input file will be different, we anticipate Round-1 systems that submitted can be used without further adjustments to submit on Round-2. All are welcome to participate in Round-2. Registration or participation in the Round-1 pilot are not prerequisites for signing up for and participating in the Round-2 pilot.

What

The Code task is to automatically generate high-quality tests for python code given a textual specification of the task the python code is intended to carry out. There will be two prompt conditions: fixed prompt and custom prompt. Participants will be given a fixed prompt for each trial_id, this fixed prompt will already have the textual specification incorporated. The submission must contain an output that used the fixed prompt. Participants will have the opportunity to customize the prompts, and will be allowed to submit up to 9 alternative custom prompts in a submission. Both the fixed prompt and 1 custom prompt per trial_id are required in a submission. These text specifications may include a method header with all the parameter names and a brief summary of what the code inputs and outputs are and the function of the code. For more details, please see the GenAI Code Pilot Evaluation Plan.

Who

Teams from academia, industry, and other research labs are invited and encouraged to contribute to Generative AI research through the GenAI platform. This platform is designed to support various modalities, including both "Generators" and "Discriminators" technologies. The GenAI Code specifically focuses on Generators and measuring the quality of AI-generated test code.

How

To take part in the GenAI Code evaluation, participants must register on this website and agree to the license to download data and upload submissions. NIST will make all necessary data resources available. Please refer to the published schedule for data release dates. Participants will be able to upload their AI-generated test code to the challenge website and see their results displayed on the scoreboard.

Task Coordinator

If you have any questions, please email to the NIST GenAI team