Cover not available

Article published In: Information Visualization
Edited by Isabel Meirelles, Marian Dörk and Yanni Loukissas
[Information Design Journal 27:1] 2022
► pp. 21–34

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB

Surprise machines

Revealing Harvard Art Museums’ image collection

ORCiD logo with link | University of Groningen
ORCiD logo with link | Harvard University
ORCiD logo with link | Yale University
ORCiD logo with link | Harvard University
 | Harvard University
ORCiD logo with link | Fachhochschule Potsdam
ORCiD logo with link | Harvard University
 | Harvard University
 | Harvard University
Open Access logo
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Published online: 10 November 2022
DOI logo with link
https://doi.org/10.1075/idj.22013.rod
Abstract
Surprise Machines is a project of experimental museology that sets out to visualize the entire image collection of the Harvard Art Museums, with a view to opening up unexpected vistas on more than 200,000 objects usually inaccessible to visitors. The project is part of the exhibition organized by metaLAB (at) Harvard entitled Curatorial A(i)gents and explores the limits of artificial intelligence to display a large set of images and create surprise among visitors. To achieve this feeling of surprise, a choreographic interface was designed to connect the audience’s movement with several unique views of the collection.
Article outline
  • 1.Introduction
  • 2.Harvard Art Museums
  • 3.Can machines curate?
  • 4.How to map 200,000 images
  • 5.Designing a post-pandemic, choreographic interface
  • 6.A substantial drawback
  • 7.Conclusions
  • Bibliography
References (50)
Bibliography
American Museum of Natural History– American Museum of Natural History. (2022). Collectionscope [JavaScript]. [URL] (Original work published 2020)
Barabási, A.-L., Bello, M., Kluge-Fabényi, J., Forde, K., Készman, J., Meirelles, I., Ratti, C. G., Ritchie, M., & Szántó, A.– Barabási, A.-L., Bello, M., Kluge-Fabényi, J., Forde, K., Készman, J., Meirelles, I., Ratti, C. G., Ritchie, M., & Szántó, A. (2020). Hidden patterns: Visualizing networks at BarabásiLab (A. Stang & P. Weibel, Eds.). Hatje Cantz Verlag.Google Scholar logo with link to Google Scholar
Benedetti, A.– Benedetti, A. (2022, January 5). Browsing and visualizing collections of images. [URL]
Birkle, C., & Däwes, B.– Birkle, C., & Däwes, B. (2019). “Old media don’t go away, they mutate”: An interview with Jeffrey Schnapp. Amerikastudien/American Studies, 64 (1), 111–125. Google Scholar logo with link to Google Scholar
Bludau, M.-J., Dörk, M., & Heidmann, F.– Bludau, M.-J., Dörk, M., & Heidmann, F. (2021). Relational perspectives as situated visualizations of art collections. Digital Scholarship in the Humanities, 36 (Supplement_2), ii17–ii29. Google Scholar logo with link to Google Scholar
Börner, K., Maltese, A., Balliet, R. N., & Heimlich, J.– Börner, K., Maltese, A., Balliet, R. N., & Heimlich, J. (2016). Investigating aspects of data visualization literacy using 20 information visualizations and 273 science museum visitors. Information Visualization, 15 (3), 198–213. Google Scholar logo with link to Google Scholar
Bostock, M., Ogievetsky, V., & Heer, J.– Bostock, M., Ogievetsky, V., & Heer, J. (2011). D3: Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics, 17 (12), 2301–2309. Google Scholar logo with link to Google Scholar
Champion, E. M.– Champion, E. M. (2016). Digital humanities is text heavy, visualization light, and simulation poor. Digital Scholarship in the Humanities, i25–i32. Google Scholar logo with link to Google Scholar
Crawford, K., & Paglen, T.– Crawford, K., & Paglen, T. (2019). Excavating AI: the politics of images in machine learning training sets. AI Now Institute. [URL]
Crockett, D.– Crockett, D. (2019). IVPY: iconographic visualization inside computational notebooks. International Journal for Digital Art History, 4 1, 3.60–3.79. Google Scholar logo with link to Google Scholar
Cuno, J. B.– Cuno, J. B. (Ed.). (1996). Harvard’s art museums: 100 years of collecting. Harvard University Museums.Google Scholar logo with link to Google Scholar
Danchilla, B.– Danchilla, B. (2012). Beginning WebGL for HTML5. Apress. Google Scholar logo with link to Google Scholar
Derry, L., Kruguer, J., Muelle, M., & Schnapp, J.– Derry, L., Kruguer, J., Muelle, M., & Schnapp, J. (2022). Designing a choreographic interface during covid-19. Movement and Computing Conference. Google Scholar logo with link to Google Scholar
Diagne, C., Barradeau, N., & Doury, S.– Diagne, C., Barradeau, N., & Doury, S. (2018). T-SNE Map. Experiments with Google. [URL]
DiMaggio, P., & Hargittai, E.– DiMaggio, P., & Hargittai, E. (2001). From the “digital divide” to “digital inequality”: Studying internet use as penetration increases. Center for Arts and Cultural Policy Studies, Princeton University.Google Scholar logo with link to Google Scholar
Drucker, J.– Drucker, J. (2013). Performative materiality and theoretical approaches to interface. Digital Humanities Quarterly, 7 (1). [URL]
Duhaime, D.– Duhaime, D. (2021). PixPlot. Yale Digital Humanities Lab. [URL] (Original work published 2017)
Foster, H.– Foster, H. (2011). The art-architecture complex. Verso.Google Scholar logo with link to Google Scholar
Geismar, H.– Geismar, H. (2018). Museum object lessons for the digital age. UCL Press. Google Scholar logo with link to Google Scholar
Harvard Art Museums– Harvard Art Museums. (2012). International Image Interoperability Framework at Harvard University. [URL]
Impett, L., & Moretti, F.– Impett, L., & Moretti, F. (2017). Totentanz. Operationalizing Aby Warburg’s Pathosformeln. New Left Review, 107 1, 68–97.Google Scholar logo with link to Google Scholar
Jacomy, M., Venturini, T., Heymann, S., & Bastian, M.– Jacomy, M., Venturini, T., Heymann, S., & Bastian, M. (2014). ForceAtlas2, a continuous graph layout algorithm for handy network visualization designed for the Gephi software. PLoS ONE, 9 (6), e98679. Google Scholar logo with link to Google Scholar
Jodidio, P.– Jodidio, P. (2014). Piano: Renzo Piano building workshop, complete works, 1966-today. Taschen.Google Scholar logo with link to Google Scholar
Karjus, A., Solà, M. C., Ohm, T., Ahnert, S. E., & Schich, M.– Karjus, A., Solà, M. C., Ohm, T., Ahnert, S. E., & Schich, M. (2022). Compression ensembles quantify aesthetic complexity and the evolution of visual art. Google Scholar logo with link to Google Scholar
Kenderdine, S., Mason, I., & Hibberd, L.– Kenderdine, S., Mason, I., & Hibberd, L. (2021). Computational archives for experimental museology, 3–18. Google Scholar logo with link to Google Scholar
Klinke, H.– Klinke, H. (2021, October 28). V-lab workshop: Visual analysis in cultural data – image plots and t-SNE maps made easy. History of Art Department. [URL]
Kräutli, F.– Kräutli, F. (2016). Visualising cultural data: Exploring digital collections through timeline visualisations. Royal College of Art.Google Scholar logo with link to Google Scholar
Latour, B.– Latour, B. (1988). The pasteurization of France (A. Sheridan & J. Law, Trans.; English edition). Harvard University Press.Google Scholar logo with link to Google Scholar
Lima, M.– Lima, M. (2011). Visual complexity: Mapping patterns of information. Princeton Architectural Press.Google Scholar logo with link to Google Scholar
Maaten, L. van der, & Hinton, G.– Maaten, L. van der, & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9 (86), 2579–2605. [URL]
Maizels, M., & Qiu, C.– Maizels, M., & Qiu, C. (Eds.). (2020). Curatorial A(i)gents. metaLAB (at) Harvard. [URL]
Manovich, L.– Manovich, L. (2008). Data visualization as new abstraction and anti-sublime. In B. Hawk, D. M. Rieder, & O. O. Oviedo (Eds.), Small tech: The culture of digital tools. University of Minnesota Press.Google Scholar logo with link to Google Scholar
Manovich, L.– Manovich, L. (2020). Cultural analytics. MIT Press. Google Scholar logo with link to Google Scholar
Manzini, E.– Manzini, E. (2016). Design culture and dialogic design. Design Issues, 32 (1), 52–59. Google Scholar logo with link to Google Scholar
McCully, E. A.– McCully, E. A. (2019). Dreaming in code: Ada Byron Lovelace, computer pioneer. Candlewick Press.Google Scholar logo with link to Google Scholar
McInnes, L., Healy, J., & Melville, J.– McInnes, L., Healy, J., & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for dimension reduction. ArXiv.Org, stat.ML. [URL]
metaLAB– metaLAB. (2022). MetaLAB (at) Harvard & FU Berlin. [URL]
Moon, C. Y. E., & Rodighiero, D.– Moon, C. Y. E., & Rodighiero, D. (2020). Mapping as a contemporary instrument for orientation in conferences. Atti Del IX Convegno Annuale AIUCD. La Svolta Inevitabile: Sfide e Prospettive per l’informatica Umanistica. Google Scholar logo with link to Google Scholar
O’Shea, K., & Nash, R.– O’Shea, K., & Nash, R. (2015). An introduction to convolutional neural networks. ArXiv:1511.08458 [Cs]. [URL]
Pietsch, C.– Pietsch, C. (2022). VIKUS viewer [JavaScript]. [URL] (Original work published 2018)
Rodighiero, D.– Rodighiero, D. (2021a). Mapping affinities: Democratizing data visualization (Open-access English edition). Métis Presses. Google Scholar logo with link to Google Scholar
Rodighiero, D.– Rodighiero, D. (2021b, August 18). Ars memorativa as the genesis of information design: A conversation with Manuel Lima. Nightingale. [URL].
Rodighiero, D., Wandl-Vogt, E., & Carsenat, E.– Rodighiero, D., Wandl-Vogt, E., & Carsenat, E. (2021). Making visible the invisible work of scientists during the Covid-19 pandemic. Visual Culture Studies, 2 1. Google Scholar logo with link to Google Scholar
Rodighiero, D., Wandl-Vogt, E., & Carsenat, E.– Rodighiero, D., Wandl-Vogt, E., & Carsenat, E. (2022). A visual translation of the pandemic. Leonardo, 55 (3). Google Scholar logo with link to Google Scholar
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., & Fei-Fei, L.– Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., & Fei-Fei, L. (2015). Imagenet large scale visual recognition challenge. ArXiv:1409.0575 [Cs]. [URL].
Seguin, B.– Seguin, B. (2018). The Replica project: Building a visual search engine for art historians. XRDS: Crossroads, The ACM Magazine for Students, 24 (3), 24–29. Google Scholar logo with link to Google Scholar
Steward, J.– Steward, J. (2021). API documentation. Harvard Art Museums. [URL] (Original work published 2015)
Turing, A. M.– Turing, A. M. (1950). Computing machinery and intelligence. Mind, LIX (236), 433–460. Google Scholar logo with link to Google Scholar
Vane, O.– Vane, O. (2019). Timeline design for visualising cultural heritage data. Royal College of Art.Google Scholar logo with link to Google Scholar
Weaver, W.– Weaver, W. (1948). Science and complexity. American Scientist, 36 (4), 536–544. JSTOR. [URL]
Cited by (5)

Cited by five other publications

Felsing, Ulrike & Max Frischknecht
2025. Visuelle Zugänge gestalten: Neue Nutzungspraktiken digitaler Archive im Kontext künstlicher Intelligenz. In Designforschung und KI [Designforschung – Designwissenschaft - Designtheorie, ], ► pp. 179 ff. DOI logo
Takahira, Kentaro, Yue Yu, Takanori Fujiwara, Ryo Suzuki & Huamin Qu
2025. Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, ► pp. 1 ff. DOI logo
Ambalavan, Ayisha Saniya & Ashwarya Chauhan
2024. AI Implementation in Art Museums Through Sensory Design. KnE Social Sciences DOI logo
Dhaenens, Laurens & Frederik Truyen
2024. Art in circulation: Creating content and context for digital reproduction of artworks. Curator: The Museum Journal 67:3 ► pp. 603 ff. DOI logo
Yang, Yuchen & Linyida Zhang
2023. Proceedings of the 5th Workshop on analySis, Understanding and proMotion of heritAge Contents, ► pp. 5 ff. DOI logo

This list is based on CrossRef data as of 11 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue