2026 GenAI Image Watermark

Watermark Embedders Transformations and Metrics

Applied Transformations

• The NIST GenAI team will evaluate the robustness of submitted watermarked images by applying a set of natural transformations representative of common real-world processing. These transformations are not designed to be adversarial; rather, they reflect typical operations that images undergo during storage, sharing, editing, and display.
  Each submitted watermarked image will be subjected to one or more transformations,including different strength levels, drawn from standardized categories, which may include (but are not limited to):
  • Compression and re-encoding (e.g., JPEG/WebP at varying quality levels)
  • Resizing and resampling (downscaling, upscaling, interpolation changes)
  • Geometric modifications (cropping, minor rotation, padding)
  • Photometric adjustments (brightness, contrast, gamma, color shifts)
  • Noise and filtering (Gaussian noise, blur, denoising)
  • Image filters (social media enhancers)
  • Format and pipeline effects (color space conversion, re-saving workflows)
• In addition to individual transformations, selected multi-step pipelines (e.g., resize → compress → crop) may be applied to better approximate realistic processing scenarios such as social media sharing or image editing workflows.
• Following transformation, watermark robustness will be assessed using the results from the participant-provided detection and decoding components. Evaluation will measure:
  • Imperceptibility (watermarked image quality)
  • Watermark detectability (presence/absence)
  • Payload recovery accuracy (for multi-bit watermarking)
  • Payload integrity (correctness of recovered bits)

Summary Of Metrics

• The embedder track will focus on three primary dimensions: perceptual similarity, generation quality and diversity (for prompt-based methods), and robustness under natural transformations.
  • Perceptual Similarity: measures the visual quality of watermarked images relative to their original or reference images.
    • For post-hoc watermarking methods (i.e., watermarking applied to provided images), textual prompt-based generation (generating pairs of watermarked and unwatermarked images), and image-based regeneration (systems that regenerate a provided image while embedding the watermark), imperceptibility will be evaluated by comparing the submitted image to the provided reference image using:
      • Peak Signal-to-Noise Ratio (PSNR)
      • Structural Similarity Index Measure (SSIM)
      • Learned Perceptual Image Patch Similarity (LPIPS and R-LPIPS)
  • Generation Quality and Diversity: for methods that generate multiple images per input prompt, diversity will be evaluated to measure variation across generated outputs while maintaining semantic consistency.
    • CLIP-based diversity metrics will be used to quantify variation among generated images for a given prompt.
    • Other metrics as appropriate
  • Robustness: a standardized set of natural (non-adversarial) transformations will be applied to the submitted watermarked images. Following transformation, each image will be processed using the participant-provided watermark detector and decoder.
    • Watermark Detectability Metrics:
      • True Positive Rate (TPR)
      • False Positive Rate (FPR) - utilizing unwatermarked images
      • Receiver Operating Characteristic Area Under Curve (ROC-AUC)
      • Brier Score
    • Payload Decodability Metrics: (for multi-bit watermarking)
      • Bit Error Rate (BER)
      • Bit Accuracy

Watermark Embedders Submission Process

• Each team will receive a tarball `wm_submission.tar.gz` containing the resources needed to implement their decoding environment, including all necessary information for their watermarks, such as the text prompt and topic files. The provided workspace will resemble the following
  
  

                          ├── configs/
                             ├── pipeline_config.yml
                             ├── submission_validator.py
                             ├── xml_parser.py
                             └── xml_utils.py
                          ├── datasets/
                             ├── watermarked/
                             ├── unwatermarked/
                          ├── inputs/
                             └── instructions.xml
                          ├── outputs/                          # To be populated by participant
                             ├── submission.xml
                             └── results.csv
                          ├── watermarks/
                             ├── watermark.py           # Example embedding script to be created by participant
                             └── decode.py              # Example decoding script to be created by participant
                             └── key.pt                 # Example key to be created by participant (if applicable)
                          └── Dockerfile
                      

• The datasets/watermarked/ folder should be populated for all watermarking method types. The datasets/unwatermarked/ folder may be empty for post-hoc and image-based generation, but must be populated for text-based generation. The watermarks/watermark.py and watermarks/decode.py files shown above are examples demonstrating how to interact with the `watermarks` module. Teams may adapt, replace, or restructure this module. NIST will mount output files to the scoring container. Therefore, teams are required to store all generated outputs, including submission.xml and results.csv, in the outputs/ directory so they can be read without errors.
• For the final submission, teams must upload a single compressed folder via the web platform This folder must contain the following two components:
  1- A datasets/ folder containing the images, organized according to the directory structure mentioned above (e.g., datasets/watermarked and/or datasets/unwatermarked).
  2- The Docker image tarball, which must be named exactly `watermark-decode.tar.gz`.
  
  The submission container must execute non-interactively, reading images from the mounted dataset volume as input and writing the results (e.g. outputs.csv) directly to the mounted output directory.
• The decoding environment must be entirely self-contained. Ensure all dependencies, libraries, and model weights are installed during the Docker build process. Since the image will not have internet access at runtime, any models usually downloaded via Python must be pre-embedded in the image.
• The detection software need to output a csv file with the following columns:
  • Filename/ID as string variable
  • Payload type (“multi-bit” | “zero-bit”) as string variable
  • Original payload length as integer value (0|32|64|128|256)
  • Extracted payload as string variable (zero-bit teams return empty strings "")
  • Watermark detection score as float value (between 0.0 to 1.0, where higher values indicates a greater probability that a watermark is present)
  • Payload extraction score (between 0.0 and 1.0, where higher values indicate greater probability that the extracted payload is correct)
• The decoding software should be designed to take images as input and return a single .csv file with the aforementioned contents and appropriate column headers, as seen in the example below. After applying the transformations, corrupted images will be transferred to the participant’s containerized implementation and NIST will use the decoding script to extract the information necessary to proceed with the evaluation. Participants are expected to provide instruction on how to run the decoding script. For consistency, please name the .csv file 'results.csv'.

  Filepath	Payload Type	Payload Length	Extracted Payload	Watermark Detection Score	Payload Extraction Score
  "datasets/topic_1…"	"multi-bit"	32	"0111000..."	0.825	0.6
  "datasets/topic_2…"	"multi-bit"	32	"1001010..."	0.783	0.75