2025 GenAI Code

Round-2 Updates

In Round-2, we updated the code bank file to refine the problem specifications so that outputs for edge case inputs are no longer ambiguous. In addition, we now allow an increased limit of 10 submissions per 24-hour period per team. Please see Section 3.4, Round-2 Pilot Updates, of the GenAI Code Pilot Evaluation Plan.

Code Generation Instructions

GenAI Code Participants are provided the input json file containing the code bank input specifications, separated and labelled by trial_id, and the corresponding fixed prompts. With this input file, GenAI Code Participants will submit a json file of test outputs containing pytest tests that can test arbitrary implementations of the methods specified by the input specifications. This json file will have outputs separated and labelled by trial_id and by prompt_number. All tests generated using the fixed prompts will have prompt_number 0. Please see the GenAI Code Pilot Evaluation Plan for detailed instructions.

Only GenAI Code participants who have completed and submitted all required forms will be allowed access. Please check the published schedule for testing data. release dates.

Optional Dry-Run Track

In addition to the main pilot track, there is a dry-run track on a small development data set. For this development data set, participants will be given not just the input problem file but also the key file, one baseline submission, and correct and incorrect code implementations. Participants may submit to the dry-run track and those submissions will be scored; however, the dry-run submissions will neither be analyzed nor scored on a scoreboard. All submissions to the dry-run track are optional.

Example Fixed Prompt

"We have python code that implements the following specification.
      Specification:
        def add(x: int, y: int) -> int:
        """
        Given two integers x, and y, return the sum of x and y. If either x or y is not
        an integer, raise a TypeError Exception.
        """

      Please write python pytest test code that comprehensively tests the code for method add to determine if the code correctly meets the specification or not. When writing tests:
      * write a comprehensive test suite,
      * test edge cases,
      * only generate tests that you are confident are correct, and
      * include tests for TypeError cases.
      Please write '###|=-=-=beginning of tests=-=-=|' before the tests. Write '###|=-=-=end of tests=-=-=|' immediately after the tests. Import any needed packages, including pytest. Additionally import the code being tested by adding the line `from genai_code_file import add` the line after '###|=-=-=beginning of tests=-=-=|'. Do not provide an implementation of the method add with the tests."

Submission Guidelines

• Please refer to the GenAI Code Pilot Evaluation Plan for specific instructions. Please see the dry-run submission provided with the data for an example submission format.
• Code should be compilable so that it can be run with the pytest package.
• Each submission should be submitted as a single .json file, and each file should contain both the fixed prompt and custom prompt submissions.
• For all submissions and all problems, the prompt used must be provided as metadata within thesubmission.
• Submission notes: according to the published schedule, the submission page (form) will be open and available (via the GenAI website) for teams to submit their data outputs. Please make sure to follow the schedule and submit on time as extending the submission dates may not be possible.
• Upon submission, NIST will validate the data outputs uploaded and report any errors to the submitter.
• Please take into consideration that submitting your data outputs indicates and assumes your agreement to the "Protocol and Rules" section of the Evaluation Plan.

Example Sample Output:

from genai_code_file import add
      import pytest
      class TestCode(object):
      def test_add(self):
        assert add(2, 3) == 5
        with pytest.raises(TypeError):
            add("abc", "def")

Submission Validation

• NIST will provide the scoring software with the validator script to participants to validate their output json files format as well as content specific to the task guidelines (e.g. required attributes). The submission platform will run this validator and fail any submissions that do not pass this validator.
• Provided with the validator scripts are the scoring scripts (to allow participants to locally score dry-run submissions) and optional utility scripts that may help with certain components.