How computer science can help art historians

I was the lead of a team of computer scientists at Rutgers that published a paper this past August titled, "Toward Automated Discovery of Artistic Influence." In that paper we reported on our research, where we used object recognition techniques to compare paintings and identify similarities between them in color, form, texture ,and content. The goal was to classify painting styles and determine how artists may have been influenced by their predecessors.

Reactions from art historians, critics, and artists ranged from enthusiastic to dismissive. London’s Daily Telegraph reported on our submission with the somewhat hyperbolic headline: "Could computers put art historians out of work?" Shortly thereafter, Professor Griselda Pollock, an art historian at University of Leeds, was more critical, writing a piece titled, "Computers can find similarities between paintings — but art history is about so much more."

It’s clear that there’s a fundamental misunderstanding in the art history community about why a computer vision scientist would be interested in exploring resemblances among works of art.

Subscribe to The Week

Escape your echo chamber. Get the facts behind the news, plus analysis from multiple perspectives.

SUBSCRIBE & SAVE

Sign up for The Week's Free Newsletters

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

Sign up

For some background: in our research, we analyzed roughly 1,700 digitized images of paintings by 66 masters from the 15th century to the 20th century.

Our computers noted interesting parallels. For example, Frederic Bazille’s Studio 9 Rue de la Condamine (1870) and Norman Rockwell’s Shuffleton’s Barber Shop (1950) possess strikingly similar compositions.

Using the results, we created a graphic of the 66 artists used in our study — what we called a "map of artists" — a graphic of artists’ names where proximity implies similarity. The computer also hypothesized about artists whose paintings may been influenced by a predecessor, like Degas and Delacroix. In many cases, these hypotheses matched those of art historians.

Computer generated "map of artists," where proximity implies similarity. (Ahmed Elgammal/The Conversation US)

In her article, Professor Pollock explained that art history is not about tracing influences by comparing form, texture, and color — what, she noted, is called "connoisseurial art history," pioneered in the 19th century by Giovanni Morelli. Professor Pollock went on to (correctly) point out that the discipline of art history has evolved since then: it encompass politics, economics, literature, ideology, theology, and philosophy, while embracing the perspectives of contemporary social movements.

Essentially, she was saying that our research was predicated on a definition of art history that’s outdated.

However we didn’t make any claims about what art history is or isn’t. We understand that influences between artists is not just about visual similarity — that, as a discipline, art history requires studying social, personal, ideological, and historical contexts.

Here’s what we find compelling: when people look at paintings, they can make sophisticated inferences beyond simply recognizing a tree, a chair, or the Christ; they can predict which artist likely did the work, note the painting’s style, its genre, and its period. Learning and judging such complex visual concepts is an impressive ability of human perception.

Such ability intrigues researchers in my discipline. It’s our ultimate goal to make a machine that passes the traditional Turing test (when an interrogator is unable to guess which of two players is a machine and which is a human). Of course in the case of paintings we can think of a similar test, which I would call the "visual Turing test." While passing this test may be a ways off, our results offer an important breakthrough: proof that a machine can learn to discover interesting similarities among paintings that humans can confirm.

Furthermore, Professor Pollock argued that it’s a fallacy to claim that computers can notice things that humans cannot. Computer vision promises — and in fact has proven — that it can overcome human vision’s spectral limitation and quantitative imprecision. In the domain of art, computer vision can offer a level of precise and objective analysis.

It’s true that influence discovery might be based on a century-old view of art history. But when a child is learning to walk, she first crawls before taking her first steps. Later, she runs.

Artificial intelligence is still in its infancy, especially as it tries to tackle challenges of increasing complexity. If we hope to make a machine understand sophisticated connections between artists, it’s only natural to initially approach the challenge the same way Giovanni Morelli and connoisseurial art historians did a century ago.

The humanities are becoming digitized; paintings, photographs, letters, centuries-old books, volumes of journals, and magazines — all are increasingly available online.

And perhaps there will be a day when the technology could evolve to look at the historical, social, and personal context of art — a day when computers could mine these vast stores of heterogeneous data to conduct an analysis of artistic influences that goes beyond the connoisseurial approach.

At that point, it would be up to art historians to figure out how to best utilize these technological advances.

Do we hope to put art historians out of work? Hardly. If anything, I tend to agree with a Nov. 10 Washington Post article headline about our research: "Computer Science Putting Art Analysis on Faster Track."

More from The Conversation US...

• Personal privacy is eroding as consent policies of Google and Facebook evoke ‘fantasy world’
• Lima climate agreement — experts react
• Why the message of the Hunger Games films is dangerous