Skip to content
FAIRSCAPE

AI-Readiness grader

Grades a dataset’s metadata against the Rubric for Review of AI-readiness Evaluation Criteria: 28 criteria in seven domains (FAIRness, Provenance, Characterization, Pre-model Explainability, Ethics, Sustainability, Computability), each scored 0, 1 or 2. The input can be an RO-Crate, a Croissant file or plain schema.org JSON-LD.

It is part of the view and assess step, next to fairscape-artifacts, which uses it for the AI-Ready section of the datasheet.

Install

Terminal window
pip install git+https://github.com/fairscape/AIreadiness-grader

Example

Terminal window
fairscape-evidence path/to/crate -o review
  • Directoryreview/
    • ai-ready-review.html review page: evidence and an automated estimate per criterion
    • ai-ready-evidence.json the same evidence as data

The input can also be a single JSON-LD file, a URL, kaggle:owner/slug or hf:org/name.

What a review looks like

This is the review for a CM4AI proteomics dataset. Each of the 28 criteria gets the evidence the grader found in the metadata and an automated estimate. Gates are marked: here 0.a scores 1 because the crate’s ARK doesn’t resolve to a recognised sustainable repository, so the result is Gating FAIL even though most domains score well.

Automated estimate per criterion EndoTag AP-MS Profiling of Chromatin Modifier Interactome Rewiring in MDA-MB-468 Cells Upon Paclitaxel Perturbation
Gating FAIL · 0.a scored 1, needs 2; 1.c scored 0, needs 1
FAIRness 0.a 0.b 0.c 0.d 5/6
Provenance 1.a 1.b 1.c 1.d 4/8
Characterization 2.a 2.b 2.c 2.d 2.e 5/6
Pre-model Explainability 3.a 3.b 3.c 4/6
Ethics 4.a 4.b 4.c 4.d 2/2
Sustainability 5.a 5.b 5.c 5.d 6/6
Computability 6.a 6.b 6.c 6.d 3/4
0 1 2 N/A gating criterion below gate
Rubric for Review of AI-readiness Evaluation Criteria, v1.8 (2026-09-10). Estimates come from the metadata alone; a person or an LLM makes the final call. See the full review, which lists the evidence behind every cell.
AI-Readiness review: EndoTag AP-MS profiling under paclitaxel Open full page
The full review page. Expand a criterion to see the questions the rubric asks, the evidence found, and what would raise the score.

Commands

CommandWhat it does
fairscape-evidenceCollects the evidence for every criterion and writes the review page. Needs no model.
fairscape-gradeHas an LLM score every criterion (example below).
fairscape-improveWrites an offline form listing the missing properties, easiest first, with a live score estimate. Download the improved ro-crate-metadata.json when you’re done.
Terminal window
fairscape-grade path/to/crate grading --model anthropic:claude-sonnet-5 --api-key $ANTHROPIC_API_KEY

Details

  • Scoring. The overall score is the mean of the seven domain scores. Some criteria are gates (for example, FAIRness 0.a must score 2). A failed gate marks the result Gating FAIL, but the score is still reported.
  • Models. --model takes any pydantic-ai model string (anthropic:, openai:, google:, groq:). Any OpenAI-compatible server works through openai: with OPENAI_BASE_URL set.
  • Offline. Pass --no-network to skip URL and registry lookups.

Source: github.com/fairscape/AIreadiness-grader