The museum catalogue problem AI keeps getting wrong—and a new model's proposed fix

A museum may photograph a carved object, attach a description and ask an AI system to help catalogue it. The difficult part is not always recognising what appears in the picture. It is getting several labels to agree. An artefact might be assigned a plausible material, period, technique and object class when each field is scored separately, yet the combination may contradict the museum's own taxonomic structure. It is the digital equivalent of filing a book in two mutually incompatible sections of a library. New research published on 1 November in Scientific Reports takes that consistency problem seriously.[1]

Feng Luo, Qi Lu and Chunyang Meng propose a system that aligns visual and textual evidence while predicting categories through hierarchical relationships. Their design uses a vision transformer to extract image features and a BERT-based encoder for text. A contrastive learning objective puts related descriptions and visual patterns closer together. A gating mechanism combines the modalities, and hierarchical classification heads predict attributes at more than one level. The novelty is that the system also penalises contradictions between parent and child classifications during training.[1]

Why the labels have to form a tree

Imagine an object described as textile, then silk, then a specific weaving technique. If a model predicts 'ceramic' at the top level and 'silk brocade' underneath, the individual words may each have seemed plausible in isolation, but the final record is structurally nonsensical. Museum databases contain precisely these kinds of linked categories, and a search or preservation programme can be undermined by inconsistent records. A museum professional must be able to navigate from a broad class to a defensible subtype, not receive an unconnected bag of keywords.[1]

The researchers introduce two ways of limiting the problem. During training, a hierarchical consistency loss discourages incompatible label distributions across levels of a taxonomy. At inference time, constrained decoding prevents the final model output from following an impossible path through the known classification tree. The second step can enforce logical compatibility even when the model is uncertain about an object's identity. It is an example of using explicit structure alongside statistical pattern recognition rather than relying on a model to remember every rule implicitly.[1]

On the Metropolitan Museum of Art Open Access dataset, the paper reports an average hierarchical F1 score of 85.2%, 6.9 percentage points above its strongest multimodal baseline. It also reports a learned hierarchy-violation rate of 2.8%, with constrained decoding eliminating the remaining structural violations at inference. These are benchmark results under the authors' specified evaluation. An F1 score combines precision and recall; 85.2% F1 is not a claim that 85.2% of every museum object's history has been verified or that curators can accept the outputs without review.[1]

The team also tested transfer on the SILKNOW silk-textile corpus and reports that ten epochs of fine-tuning exceeded the comparison baselines in the evaluated setting. That result matters because one museum collection can differ enormously from another in photography, catalogue vocabulary, language and how its objects have been classified historically. A model that only succeeds with familiar labels has limited value to institutions with scarce digitisation resources. Transfer results are encouraging, but the exact collection and evaluation conditions still matter.[1]

Correct categories do not guarantee correct history

The more consequential boundary is between taxonomy and provenance. A structurally valid record can still be historically wrong. A photograph may not reveal a later restoration, a disputed origin, an inscription hidden from view or uncertainty about how an item entered a collection. Similarly, a constrained classifier might force an object into the closest recognised category when the collection's taxonomy lacks the right one. Constrained outputs are internally legitimate; they are not an independent historical verdict.[1]

For a real catalogue, this suggests a division of labour. The machine could propose consistent material and type labels, identify comparable objects and flag records that violate known hierarchy rules. A curator would need to check uncertain classifications against the physical object, accession documentation, expert literature and collection-specific conventions. The model should preserve confidence and the provenance of its own suggestions so that corrections do not become anonymous database facts.[1]

The research offers a useful alternative to AI's usual promise of endlessly generating text about images. Here, intelligence is partly the ability to say that two labels cannot be true in the catalogue's own system. When institutions hold millions of records that were created by different generations of staff, consistency is valuable. The challenge is to gain that computational order without flattening the ambiguity and contested history that make cultural objects worth studying.[1]

Report an error or suggest a correction