PAST CHALLENGE
2025 GenAI Image
GenAI Image Challenge evaluating Generation and Discrimination capability.
Image Discriminators Overview
The primary goal of the GenAI pilot is to understand system behavior detecting AI-generated vs. human-generated content.
The Image-D task is a detection task focused on determining if a target output image was generated using Generative AI models or humans.
For each Image-D trial consisting of a single image, the Image-D detection system must render a confidence score with a higher number indicating a higher likelihood that the target image was generated using AI models. The primary metric for measuring detection performance will be the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC), as well as the Equal Error Rate (EER), True Positive Rate (TPR) at a given False Positive Rate (FPR), and Brier score values for targets (AI-generated images) and non-targets (human images).
The GenAI pilot challenge provides data (test sets) created by both G-participants and the NIST GenAI team. This allows D-participants to develop and run a system on their own hardware platform. Discriminator participants can then submit their system outputs to a web-based leaderboard, where scores and results are displayed.
The data from G-participants will only be accessible to D-participants once the G-participants submit their data packages to NIST and the NIST GenAI team approves the data. NIST will report performance measures for D-participant system outputs, displayed through a leaderboard, using the G-participants data.
Please refer to the discriminator evaluation plan for the details.
Data resources will be available for download once the registration is open and the data release has been announced. NIST will also release GenAI Scorer and Format Validator scripts.
Image Discriminator Instructions
The participants are not allowed to use the generators’ topics files by any means to extract information that can assist their systems in predicting the testing image source. Teams who participate in both tasks (Generators and Discriminators) are required to take all needed precautions to isolate the generators' topic files from discriminator's system development.
During result analysis and for presentation of results, all discriminator only teams will be in a separate category for reporting results purposes, while teams who participate as a generator and discriminator will be in a separate category.
All images must be processed independently of each other within a given round of evaluation and across all evaluation rounds, meaning content extracted from the data must not affect the processing of data from another rounds’ data.
System Input File
For a given task, a system’s input is the task index file, called <modality_id>_<dataset_id>_<task_id>_index.csv. Given an index file, each row specifies a test trial. Taking the corresponding media (eg. texts or images) as input(s), systems perform detection tasks.
The following format constitutes the index file for the D-participant system input:
genai25_Image-D_detection_index.csv
|
|
|
DatasetID |
(string) The ID of the dataset release (e.g., GenAI25-G1-Image-set1) |
|
TaskID |
(string) The globally unique ID of tasks. Tasks could be summarization, generation, translation, question-answering (e.g., Detection) |
|
FileID |
(string) The globally unique ID of the image trials (e.g., file_0001.webP) |
Example of the CSV file with delimiter “|”.
DatasetID | TaskID | FileID
GenAI25-G1-Image-set1 | Detection | file_0001.webpSystem Output File
The system output file must be a CSV file with the separator “|”. Please include the optimal cutoff (threshold) for the confidence score binary classification in your file name. For example, "cutoff-50" means the threshold is 0.5 The filename for the output file must be a user-defined string that identifies the submission with no spaces or special characters besides ‘_-.’ (e.g., `genai25_image_d_sys_model-01_cutoff-50.csv`).
The system output CSV file for the Image-D detection task must follow the format below:
genai25_image_d_sys_model-01_cutoff-XX.csv
|
||
|
DatasetID |
(string) The ID of the dataset release, e.g., GenAI25-G1-Image-set1 |
|
|
TaskID |
(string) The unique ID of the task, e.g., Detection |
|
|
DiscriminatorID |
(string) The globally unique ID of Discriminator (D) participants, e.g. D-participant_001 |
|
|
ModelVersion |
(string) The system model version on D-participant submission (e.g., MySystem_DallE) |
|
|
FileID |
(string) The globally unique ID of the image trials (e.g., file_0001.webp) |
|
|
ConfidenceScore |
(float) in the range [0,1], the larger, the more confidence that the output is AI generated |
|
Example of the CSV file with delimiter “|”.
DatasetID | TaskID | DiscriminatorID | ModelVersion |FileID | ConfidenceScore GenAI25-G1-Image-set1 | Detection | D-participant_01 | MySys_DallE | file_0001.webp | 0.7Validation
The FileID column in the system output [submission-file-name].csv must be consistent with the FileID in the <modality_id>_<dataset_id>_<task_id>_index.csv file. The row order may change, but the number of the files and file names from the system output must match the index file.
To validate your system output locally, D-Participants may use the validation script provided by NIST with the dataset release.
Submission
System output submission to NIST for subsequent scoring must be made through the web platform using the submission instructions described above. To prepare your submission, you will first make .tar.gz (or .tgz) file of your system output CSV file via the UNIX command ‘tar zcvf [submission_name].tgz [submission_file_name].csv’ and then upload the system output tar file under a new or existing ‘System’ label. This system label is a longitudinal tracking mechanism that allows you to track improvements to your specific technology over time.
Please ensure timely submission of your files to allow us sufficient time to address any transmission errors before the due date. Note that submissions received after the stated due dates for any reason will be marked late and may not be scored. Please refer to the published schedule for the details.
Please take into consideration that submitting your system outputs indicates and assumes your agreement to the Rules of Behavior.
For the best user experience with the scoreboard, please use Google Chrome.
