PAST CHALLENGE
2024 GenAI Text
GenAI evaluation on capabilities of text to text generators and discriminators.
T2T Discriminators Overview
The primary goal of the GenAI pilot is to understand system behavior detecting AI-generated vs. human-generated content.
The T2T-D task is a detection task focused on determining if a target output was generated using Generative AI or humans. The T2T-D detection task consists of detecting if a target text summary was generated based on large language models (LLMs) such as ChatGPT.
For each T2T-D trial consisting of a single summary, the T2T-D detection system must render a confidence score with a higher number indicating a higher likelihood that the target text summary was generated using LLM-based models. The primary metric for measuring detection performance will be the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC), as well as the Equal Error Rate (EER), True Positive Rate (TPR) at a given False Positive Rate (FPR), and Bayes risk for varying tradeoff values for cost of errors.
The GenAI pilot challenge provides data, including dry-run sets and test sets, created by both G-participants and the NIST GenAI team. This allows D-participants to develop and run a system on their own hardware platform. Discriminator participants can then submit their system outputs to a web-based leaderboard, where scores and results are displayed.
The data from G-participants will only be accessible to D-participants once the G-participants submit their data packages to NIST and the NIST GenAI team approves the data. However, NIST will provide pilot data generated by the NIST GenAI team for D-participants to start the development of their systems. NIST reports performance measures for D-participant system outputs, displayed through a leaderboard, using either NIST pilot data or the evolved G-participants data.
Please refer to the discriminator evaluation plan for the details.
Data resources will be available for download once the registration is open and the data release has been announced. NIST will also release GenAI Scorer and Format Validator scripts.
T2T Discriminator Instructions
System Input File
For a given task, a system’s input is the task index file, called <modality_id>_<dataset_id>_<task_id>_index.csv. Given an index file, each row specifies a test trial. Taking the corresponding media (texts or images) as input(s), systems perform detection tasks.
The following format constitutes the index file for the D-participant system input:
| genai24_T2T-D_detection_index.csv | |
|---|---|
| DatasetID | (string) The ID of the dataset release (e.g., GenAI24-NIST-pilot-T2T-D-set-1) |
| TaskID | (string) The globally unique ID of tasks. Tasks could be summarization, generation, translation, question-answering (e.g., detection) |
| FileID | (string) The globally unique ID of the text summary trials (e.g.,file_0001.txt) |
Example of the CSV file with delimiter “|”.
DatasetID | TaskID | FileID GenAI24-NIST-pilot-T2T-D-set-1 | detection | xxx_000011.txtSystem Output File
The system output file must be a CSV file with the separator “|”. Please include, in your filename, the recommended cutoff (threshold) for the confidence score for binary classification. For example, "cutoff-50" means the threshold is 0.5, and confidence scores greater than or equal to 0.5 are classified as "AI" and confidence scores less than 0.5 are classified as "Human" generated content. The filename for the output file must be a user-defined string that identifies the submission with no spaces or special characters besides ‘_-.’ (e.g., `genai24_t2t_d_sys_model-01_cutoff-50.csv`).
The system output CSV file for the T2T-D detection task must follow the format below:
| genai24_t2t_d_sys_model-01.csv | ||
|---|---|---|
| DatasetID | (string) The ID of the dataset release, e.g., GenAI24-NIST-pilot-T2T-D-set-1 | |
| TaskID | (string) The ID of the summary files, e.g., detection | |
| DiscriminatorID | (string) The site name of Discriminator (D) participants, e.g. D-NIST_Site | |
| ModelVersion | (string) The system model version on D-participant submission (e.g., mySys_GPT4.0) | |
| FileID | (string) The globally unique ID of the text summary trials (e.g., file_0001.txt) | |
| ConfidenceScore | (float) in the range [0,1], the larger, the more confidence that the output is AI generated | |
Example of the CSV file with delimiter “|”.
DatasetID | TaskID | DiscriminatorID | ModelVersion | FileID | ConfidenceScore GenAI24-NIST-pilot-T2T-D-set-1 | detection | D-NIST_Site | MySys_GPT4.0 | file_0001.txt | 0.7Validation
The FileID column in the system output [submission-file-name].csv must be consistent with the FileID in the <modality_id>_<dataset_id>_<task_id>_index.csv file. The row order may change, but the number of the files and file names from the system output must match to the index file.
To validate system output locally, D-Participants may use the validation script provided by NIST with the dataset release.
Submission
System output submission to NIST for subsequent scoring must be made through the web platform using the submission instructions described above. To prepare your submission, you will first make .tar.gz (or .tgz) file of your system output CSV file via the UNIX command ‘tar zcvf [submission_name].tgz [submission_file_name].csv’ and then upload the system output tar file under a new or existing ‘System’ label. This system label is a longitudinal tracking mechanism that allows you to track improvements to your specific technology over time.
Please ensure timely submission of your files to allow us sufficient time to address any transmission errors before the due date. Note that submissions received after the stated due dates for any reason will be marked late and may not be scored. Please refer to the published schedule for the details.
Please take into consideration that submitting your system outputs indicates and assumes your agreement to the Rules of Behavior.
We emphasize that this is a pilot study (Round-1 Submissions on Testset-1). The primary purpose of the GenAI pilot is to develop an evaluation pipeline between the NIST team and participants. Therefore, we encourage all participants to submit their system output, regardless of their system performance.
For the best user experience with the scoreboard, please use Google Chrome.
