What is AITE?

The Artificial Intelligence Technology Evaluation (AITE) provides volunteer testing of AI models on blind data and its sequestered environment mitigates the risk of train/test data contamination to ensure rigorous, objective assessment. The infrastructure provided by NIST will provide common data, metrics and scoring to help developers understand the performance of their models. Initially, AITE will consist of tasks for three use cases (1) Quantum Dots, (2) Genomics Data, and (3) Video Events. Over time, AITE will build out multiple tasks under various themes (Quantum, Video, NLP, …). Each task will have the normal evaluation specification document, and submission protocols, and submissions will be scored and posted.

What are the benefits of providing data for testing?

Evaluating AI technology on data that is reflective of the actual data and application is essential for the measurements to be apt. However, conducting evaluations is challenging to do precisely and requires a substantial amount of infrastructure. By providing data to AITE, you receive careful measurements of top models on your data conducting your task without having to conduct the evaluation yourself.

What are the benefits of submitting a model for testing?

By submitting your model, you will learn how your model performs on an increasing number of datasets and tasks without having to coordinate your own evaluation. At the same time, you will learn how other models perform on the same data using the same metrics, thus improving comparability while also ensuring the data used for evaluation is not part of the training set for any model.

What is the schedule for testing?

Testing will begin in Summer 2026 and be ongoing. Note that AITE is being developed in four phases:

• Phase 1: (Initial phase) AITE will accept a very limited number of additional AI application tests and models for testing. The focus of this phase will be to exercise and hone the processes for working with external collaborators to develop tests and test external models.
• Phase 2: (Early growth and innovation phase) AITE will accept a small number of additional AI application tests and models for testing. The focus of this phase will be to increase the types and varieties of tasks/modalities and domains and corresponding evaluation techniques.
• Phase 3: (Accelerated growth and innovation phase) AITE will accept a large number of additional AI applica- tion tests and models for testing. The focus of this phase will be scaling the number of tests and models, as well as the evaluation techniques.
• Phase 4: (Continuous growth and innovation phase) AITE will continue to accept a large and growing number of additional AI application tests and models for testing. The focus of this phase will be on long term growth and alignment with emerging needs.

What types of models are being tested?

Any model that can adhere to the API and adheres to the evaluation criteria can be submitted for testing.

How will my data and models be protected?

The data and models submitted to AITE will be treated and protected as controlled unclassified information.

Will the results be publicly available?

Yes, the results will be made available publicly on the AITE website.

Who can participate in AITE?

Anyone who can abide by the AITE evaluation rules and requirements. This includes both the AITE Evaluation Plan as well as the AITE Participation Agreement.

How to interpret the results?

AITE currently consists of a relatively small number of datasets and tasks. Performance results on AITE tasks should not be expected to be similar to new data and tasks per se. We envision in the future having many datasets and tasks, at which point some generalization of results might be possible in some circumstances. Note: NIST does not approve, recommend, or endorse any proprietary product or proprietary material. The report that NIST creates should not be construed or represented as endorsements for any participant’s system or commercial product on the part of NIST or the U.S. Government.

What’s in it for the public?

AITE will provide infrastructure to facilitate the acceleration of AI innovation and effectiveness and its ability to improve the quality of life for everyday people.