2024 GenAI Text

Data Generation Instructions

NIST human assessors developed topics of interest. Each assessor created a topic and chose a set of 25 relevant documents. The testing dataset documents will come from a corpus comprising a set of newswire articles. NIST will distribute a subset of topics and relevant documents.

Only T2T generator participants who have completed and submitted all required data agreement forms will be allowed access. As the example below shows, each topic includes an id (num), title, and the required topic statement (narr). The “docs” tag indicates the source relevant documents to be used when generating the required summaries. Please check the published schedule for testing data release dates.

Example of topic:

                    
                        <topic>
    <num> topic_0001 </num>
    <title> North Medical Center  </title>
    <narr>
    Describe the activities of John Smith and the North Medical Center. 
    </narr>
    <docs>
    article_0000
    article_0001
    article_0002
    </docs>
</topic>
                

Submission Guidelines

• Each team may submit up to 5 runs for a data generation package. Each run should include one summary per topic.
• Each run should contain summaries for all topics (a run can not skip a topic or submit summaries for a subset of the topics). Please refer to the generator data specification for summary generation instructions.
• Summary content should be free from offensive text or inappropriate remarks. NIST has the right to exclude any summary or whole runs if the content proves to be inappropriate for the general public.
• Each run should include high-level metadata to characterize the generator system as requested by the below run format and DTD file. As explained in the DTD file, teams need to provide some required information/parameters, such as:
  • trainingData: Name of training dataset or collection of different datasets or source data
  • teamName: The name of the team as registered on the NIST GenAI website
  • priority: The priority of the submitted run (the lower number, the higher the priority). For any required manual review of submissions, NIST may need to limit effort to only the highest priority runs.
  • trained: A boolean (T or F) to indicate if the run was the output of a trained system by the team specifically for this task (T) or the output of an already existing system that the team used to generate the outputs (F)
  • desc: A high-level description of the system that generated this run
  • link: A link to the model used to generate the run (e.g. GitHub, etc)
  • topic: The topic id (the “num” field in the topic XML file)
  • elapsedTime: The processing time of the model (with hardware specs) to generate the summary after the topic and documents were given to it.

Example of a sample run:

                    
                        <!DOCTYPE GeneratorResults SYSTEM "GeneratorResult.dtd">
<GeneratorResults teamName="ExampleTeam">
  <GeneratorRunResult trainingData="OpenAI" version="1.0" priority="1" trained="T" desc="Short description about generation approach." link="https://hyperlink_to_document_source (if available)">
    <GeneratorTopicResult topic="topic_0001" elapsedTime="5">
    this is a 250-word summary of topic_0001
    </GeneratorTopicResult>
    
    <GeneratorTopicResult topic="topic_0002" elapsedTime="5">
    this is a 250-word summary of “topic_0002"
    </GeneratorTopicResult>
    
    <!-- ... -->
    <GeneratorTopicResult topic="topic_0003" elapsedTime="5">
    this is a 250-word summary of topic_0003
    </GeneratorTopicResult>
  </GeneratorRunResult>
</GeneratorResults>

Generator Data submission validation

• NIST will provide, prior to submission dates, a validator script to participants to validate their output XML files format as well as content specific to the task guidelines (e.g. topic ids, empty required attributes, etc). All generator teams should validate their runs before submitting them to NIST.
• Submission notes: according to the published schedule, the submission page (form) will be open and available (via the GenAI website) for teams to submit their data outputs. Please make sure to follow the schedule and submit on time as extending the submission dates may not be possible.
• Upon submission, NIST will validate the data outputs uploaded and report any errors to the submitter.
• Please take into consideration that submitting your data outputs indicates and assumes your agreement to the Rules of Behavior .

Note:

• AUC_DETECTOR_1 & AUC_DETECTOR_2 scores close to 0.5 represent better performance.
• Higher BRIER_DETECTOR_1 & BRIER_DETECTOR_2 scores represent better performance.

Detector_1 and Detector_2 are baseline detectors, not D- participants. Baseline detectors are not included in Round 2 D-submissions.