Kress Foundation Grant for Image Analysis Toolkit

The Frick Art Reference Library Photoarchive recently received a $25,000 grant from the Kress Foundation to support the creation of a toolkit that will perform computer vision analysis on digitized art historical photo archives. John Resig, Dean of Computer Science at Khan Academy, will carry out the project. This toolkit will be a groundbreaking application of technology that will transform the way photo archives are used. The potential of this tool as an aid to researchers in the discovery of new information is far-reaching and, in the near future, this and similar technologies will make large-scale, cross-institutional photo archive collaboration feasible. Once completed, the toolkit will be made freely available to other institutions.

The toolkit will be developed from the original program Resig designed for his online database of more than 200,000 images of Japanese woodblock prints. Recognizing the value of this work, Frick photoarchivists requested that he test this technology on the Photoarchive’s digitized images of more than 14,000 works of art attributed to anonymous Italian masters. The test analysis proved remarkably successful. Many alternate photographs of the same work of art (see example above) were identified throughout the collection. Unfortunately, it is not unusual for photographs of the same work of art to be identified under different regional schools and filed in more than one location, as was the case for these two photographs. The analysis of the image recognition technology overrode such human errors, however, and successfully unified these two reproductions. An essay discussing the preliminary analysis of the Frick Photoarchive’s anonymous Italian art can be found here.

The toolkit funded by the Kress Foundation will include the following four components:

1. Tools for performing a bulk upload of images from a photoarchive to the MatchEngine image analysis service and for downloading all the similarity results en masse. These results will be stored in a local database that can be used for future analysis, categorization, and report generation.
2. A tool for categorizing the results of the image analysis into a form that will be usable by a photo archivist for updating the catalogue. The tool will provide a way to view two similar images and the accompanying metadata, and quickly categorize the match. This will allow researchers to understand larger trends in the collection, such as relationships between stylistically similar works of art that were previously unknown or duplicate images that were cataloged separately.
3. A tool that provides detailed statistics on the discovered matches. This tool will provide a high-level view of the quality of the matches and the types of matches that were found.
4. An analysis tool for the discovery of image matches between the photographs of multiple photoarchives. The results of the analysis will be searchable, facilitating the discovery of new and interesting relationships with other archives. The Frick images have already been analyzed in combination with 38,500 digitized photographs of fifteenth-century Italian works of art in the photo archive of the Fondazione Zeri at the University of Bologna. In one case, two images in the Frick Photoarchive that have no previously noted relationship or image similarity both matched a single photograph in the Zeri archive. This revealed a previously unknown connection between the two Frick images—one that could not have been discovered using traditional analysis done within a single institution. Other similar findings in the cross-collection analysis underscored that merging multiple collections for visual analysis yields discoveries that cannot be made in referencing a single archive and could never be accomplished in an analog environment.

In order to make the toolkit useful to other institutions, documentation that explains how images and image metadata should be prepared for analysis, as well as detailed instructions on how to perform and understand the results of the analysis, will be included. Stayed tuned for more updates regarding this exciting development in the Photoarchive.

Composite image of two photographs representing a small devotional altarpiece attributed to the Italo-Byzantine School (13th century), Madonna and Child, undated. Tempera on panel, no dimensions. Unlocated.