GenAI: Image Challenge

Image Discriminator Instructions

The participants are not allowed to use the generators’ topics files by any means to extract information that can assist their systems in predicting the testing image source. Teams who participate in both tasks (Generators and Discriminators) are required to take all needed precautions to isolate the generators' topic files from discriminator's system development.

During result analysis and for presentation of results, all discriminator only teams will be in a separate category for reporting results purposes, while teams who participate as a generator and discriminator will be in a separate category.

All images must be processed independently of each other within a given round of evaluation and across all evaluation rounds, meaning content extracted from the data must not affect the processing of data from another rounds’ data.

System Input File

For a given task, a system’s input is the task index file, called <modality_id>_<dataset_id>_<task_id>_index.csv. Given an index file, each row specifies a test trial. Taking the corresponding media (eg. texts or images) as input(s), systems perform detection tasks.

The following format constitutes the index file for the D-participant system input:

genai25_Image-D_detection_index.csv
DatasetID	(string) The ID of the dataset release (e.g., GenAI25-G1-Image-set1)
TaskID	(string) The globally unique ID of tasks. Tasks could be summarization,  generation, translation, question-answering (e.g., Detection)
FileID	(string) The globally unique ID of the image trials (e.g., file_0001.webP)

Example of the CSV file with delimiter “|”.

DatasetID             | TaskID      | FileID         
GenAI25-G1-Image-set1 | Detection   | file_0001.webp

System Output File

The system output file must be a CSV file with the separator “|”. Please include the optimal cutoff (threshold) for the confidence score binary classification in your file name. For example, "cutoff-50" means the threshold is 0.5 The filename for the output file must be a user-defined string that identifies the submission with no spaces or special characters besides ‘_-.’ (e.g., `genai25_image_d_sys_model-01_cutoff-50.csv`).  

The system output CSV file for the Image-D detection task must follow the format below:

genai25_image_d_sys_model-01_cutoff-XX.csv
DatasetID	(string) The ID of the dataset release, e.g., GenAI25-G1-Image-set1
TaskID	(string) The unique ID of the task, e.g., Detection
DiscriminatorID	(string) The globally unique ID of Discriminator (D) participants, e.g. D-participant_001
ModelVersion	(string) The system model version on D-participant submission (e.g., MySystem_DallE)
FileID	(string) The globally unique ID of the image trials (e.g., file_0001.webp)
ConfidenceScore	(float) in the range [0,1], the larger, the more confidence that the output is AI generated

Example of the CSV file with delimiter “|”.

DatasetID             | TaskID    | DiscriminatorID  | ModelVersion |FileID       | ConfidenceScore
GenAI25-G1-Image-set1 | Detection | D-participant_01 | MySys_DallE | file_0001.webp |  0.7

Validation

The FileID column in the system output [submission-file-name].csv must be consistent with the FileID in the <modality_id>_<dataset_id>_<task_id>_index.csv file. The row order may change, but the number of the files and file names from the system output must match the index file.

To validate your system output locally, D-Participants may use the validation script provided by NIST with the dataset release.

Submission

System output submission to NIST for subsequent scoring must be made through the web platform using the submission instructions described above. To prepare your submission, you will first make .tar.gz (or .tgz) file of your system output CSV file via the UNIX command ‘tar zcvf [submission_name].tgz [submission_file_name].csv’ and then upload the system output tar file under a new or existing ‘System’ label. This system label is a longitudinal tracking mechanism that allows you to track improvements to your specific technology over time.

Please ensure timely submission of your files to allow us sufficient time to address any transmission errors before the due date. Note that submissions received after the stated due dates for any reason will be marked late and may not be scored. Please refer to the published schedule for the details.

Please take into consideration that submitting your system outputs indicates and assumes your agreement to the Rules of Behavior.

Data Generation Instructions

NIST GenAI team created a set of topics using a set of images covering a diverse set of attributes and categories (see sample images in specification plan). The goal of the descriptions and topics is to represent a real-world dataset of persons captured images as uploaded by general internet users. Topics will be distributed by NIST. Only GenAI generator participants who have completed and submitted all required data agreement forms will be allowed access. In total, there will be about  150 topics. As the example below shows, each topic includes an id (num), title, and the required topic prompt (textual description). All topics will be aggregated in one master xml file and released to Generator teams. Please check the published schedule for testing data release dates

Example of topic:

<topic>
  <num> topic_5445 </num>
  <title> Young Man  </title>

 <prompt> A young Asian man with glasses looking upward to the sky  </prompt>

  </topic>

Submission Guidelines

• Each team may submit up to 10 runs per evaluation round. Each run should include a maximum of 10 images per topic. A submission run should be in the format of a single compressed zip file to include the run results xml file (see below for specifications), any sample images used as part of the prompt, and all images generated for all topics. No multiple zip files are allowed per run.
• Please use the following folder naming conventions for your run submission:

  images_prompts : folder name (as applicable) to contain any images used as part of any topic. Please use the following descriptive file names format to easily identify topics that images have been part of (topic.1.prompt.1.webp, topic.1.prompt.2.webp, etc).

  images_generated : folder name to contain all images for all topics (e.g. topic.2.image.1.webp, topic.2.image.2.webp, … etc)

• Each run should contain images for all topics; a run can not skip a topic or submit images for a subset of the topics.
• All images must be submitted as webP lossless files, and follow the guidelines for aspect ratios and/or resolution allowed. Please see specification document for allowed values. EXIF metadata must be preserved in the images.
• Please remember to dedicate 1 image out of the 10 to be generated using the NIST-provided textual description. The other 9 images should all be generated using a fixed prompt built automatically (not manually).
• Modified system prompts may include sample images (i.e. final prompt is composed of text + image(s)). If a team decides to add sample images to the prompt, then a maximum number of 2 images should be used per prompt. These images are optional and a team may decide to use text prompts only. To be specific, these image samples should be fixed for all prompts within a topic (a team must use the same 1 or 2 image samples as part of the prompt for all generated images within a topic). For example, given 150 topics, the maximum number of image prompts that a team may use will be 300 images.
• Image content should be free from offensive visuals or inappropriate remarks. Prohibited images include, but are not limited to, any image with explicit nudity, sexual content, hate symbols, child exploitation, or non-consensual images. NIST has the right to exclude any image or whole runs if it determines the content to be inappropriate for the general public. The GenAI team will have a process to automatically detect image content and any flagged images will be reviewed by a team of NIST personnel for final recommendation.
• Each run should include high-level metadata to characterize the generator system as requested by the below run format and DTD file. As explained in the DTD file, teams need to provide some required information/parameters, such as:
• teamName: The name of the team as registered on the NIST GenAI website
• trainingData: Name of training dataset or collection of different datasets or source data
• priority: The priority of the submitted run (the lower number, the higher the priority). For any required manual review of submissions, NIST may need to limit effort to only the highest priority runs. Priority should be an integer with a range between 1 to 10
• trained: A boolean (T or F) to indicate if the run was the output of a trained system by the team specifically for this task (T) or the output of an already existing system that the team used to generate the outputs (F)
• desc: A high-level description of the system that generated this run
• link: A link to the model used to generate the run (e.g. GitHub, etc)
• topic: The topic id (the “num” field in the topic xml file)
• elapsedTime:The processing time of the model per topic to generate the image after the topic was presented to it.
• usedImagePrompts: A boolean (T or F) to indicate if a topic has used (T) sample images (up to 2 images) as part of the prompt in addition to text, or did not use any images (F).
• Please use the image file naming convention as demonstrated in the example below (eg. topic.1.image.1.webp, topic.1.image.2.webp, etc) to save and transmit your image files.
• prompt: The text description (prompt) used by the system to generate the corresponding image.
• NIST-prompt:A boolean to indicate if the generated image was based on the provided based on the official prompt (T), or based on a modified prompt by the team (F)

Example of a sample run:

<!DOCTYPE GeneratorResults SYSTEM "GeneratorResults.dtd">
<GeneratorResults teamName="participant_1" >
<GeneratorRunResult trainingData="OpenAI" priority="1" trained="T" desc="This run uses the top secret x-component" link="TBD">

<GeneratorTopicResult topic="1" elapsedTime="5" usedImagePrompts="F">
<Image filename="topic.1.image.1.webp" prompt="prompt_NIST_text_description" NIST-prompt="T">
<Image filename="topic.1.image.2.webp" prompt="fixed_prompt_1" NIST-prompt="T">
<Image filename="topic.1.image.3.webp" prompt="fixed_prompt_1" NIST-prompt="F">
<!-- ... -->
<!-- ... -->
</GeneratorTopicResult>

<GeneratorTopicResult topic="2" elapsedTime="5" usedImagePrompts="T">
<Image filename="topic.2.image.1.webp" prompt="prompt_NIST_text_description" NIST-prompt="T">
<Image filename="topic.2.image.2.webp" prompt="fixed_prompt_2" NIST-prompt="F">
<Image filename="topic.2.image.3.webp" prompt="fixed_prompt_2" NIST-prompt="F">
<!-- ... -->
<!-- ... -->
</GeneratorTopicResult>

<!-- ... -->

<GeneratorTopicResult topic="150" elapsedTime="150" usedImagePrompts="T">
<Image filename="topic.150.image.1.webp" prompt="prompt_NIST_text_description" NIST-prompt="T">
<Image filename="topic.150.image.2.webp" prompt="fixed_prompt_150" NIST-prompt="F">
<Image filename="topic.150.image.3.webp" prompt="fixed_prompt_150" NIST-prompt="F">
<!-- ... -->
<!-- ... -->
</GeneratorTopicResult>
</GeneratorRunResult>
</GeneratorResults>

Generator Data submission validation

• NIST will provide, prior to submission dates, a validator script to participants to validate their output XML files format as well as content specific to the task guidelines (e.g. topic ids, empty required attributes, etc). All generator teams should validate their runs before submitting them to NIST. Example of available DTD validators (via a shell script): xmllint --valid simple_sample.xml

• notes: according to the published schedule, the submission page (form) will be open and available (via the GenAI website) for teams to submit their data outputs. Please make sure to follow the schedule and submit on time as extending the submission dates may not be possible.
• Upon submission, NIST will validate the data outputs uploaded and report any errors to the submitter.
• Please take into consideration that submitting your data outputs indicates and assumes your agreement to the code of conduct & Rules of Behavior .