What Harold Cohen learnt by teaching a computer to paint

Harold Cohen could build a machine that picked up brushes and mixed dyes. Getting it to choose colours worth painting with was more troublesome. In a 1995 conversation with Becky Cohen, the artist described a frustrating discovery: decades of experience had given him judgements he could use, but struggled to explain. The robot needed those judgements made explicit.[1]

Cohen had conceived his art-making program, AARON, at the University of California, San Diego in the late 1960s, then developed it further at Stanford’s Artificial Intelligence Lab from 1973 to 1975. He encoded decisions about shapes, space and the progression of a drawing. The project grew into a long encounter between an artist’s practical knowledge and the demand that a computer be told how to act.[2]

Rules that could make a picture

In his 1979 paper, What is an Image?, Cohen described a system that began with an empty drawing and generated the information it needed as it proceeded. It kept an internal account of the developing work. One part planned forms; another allocated space and could reject a proposal if the available area was too small or the wrong shape. Lower levels worked out the movements that would produce a line.[3]

The version described there did not learn from an archive of earlier pictures. Its knowledge came through Cohen’s rules, while each emerging drawing created fresh conditions for the next decision. This helps explain how a program could generate different pictures without being handed a finished composition to trace.[3]

Think of an illustrative drawing problem: place two irregular shapes on a page, give one an opening and put a small cluster of marks nearby. Changing the shapes’ sizes and distances may make the cluster seem contained, attached or separate. A rule about proximity can affect what the whole picture appears to mean. The decisions can be modest individually and consequential together.

The Whitney’s account of AARON describes Cohen’s attention to precisely such relationships: large and small forms, areas of differing detail and objects nearer or farther apart. His study of children’s drawing and a 1973 visit to California’s Chalfant Valley petroglyphs fed his interest in how marks become images. As the program evolved, its repertoire moved from evocative forms towards more explicit figures and environments.[2]

Getting screen colour onto paper

Colour exposed another layer of knowledge. In the 1995 interview, Cohen explained that a convincing screen palette would not transfer cleanly to the dyes used by his painting machine. He could translate only about half the screen colours satisfactorily, he said. He eventually rebuilt the approach around the physical dyes, whose behaviour on paper could differ from their appearance in the pot.[1]

The machine had encountered a materials problem as well as an image problem. Specifying a colour and producing it with a particular liquid, brush and surface are different jobs. A painter can notice a disappointing result and adjust. A system working from advance instructions needs a useful account of those materials before the brush touches the sheet.

Cohen’s 1999 account identifies brightness—the lightness or darkness of a colour—as his crucial way into the problem. This was something he had taught students for years. Turning it into the foundation of a colouring system helped AARON organise palettes in which adjoining forms remained distinguishable. He described preparing roughly 1,200 dye mixtures, then developing a way to estimate how dilution would change their brightness.[4]

The intellectual reward ran both ways. Cohen wrote that programming colour brought him to a further distinction: broad colour relationships helped shape a viewer’s response, while what happened at the boundaries of forms was central to their legibility. The act of implementation was pushing his account of painting beyond the knowledge with which he had begun.[4]

This is a demanding kind of automation. A request such as ‘choose a pleasing palette’ leaves almost everything interesting unresolved. A working procedure must decide which relationships deserve attention and how one choice limits the next. The value lies partly in that interrogation of the task: every vague instruction conceals a decision someone may know how to make without knowing how to describe it.

Working inside the program’s space

In February 2016, Cohen described a touch-screen system for colouring AARON’s drawings. His finger became the brush on a seven-foot screen. The software recorded strokes through numbered colour ‘pots’, so changing a pot’s contents could alter every associated stroke without physically painting over it.[5]

That representation changed the activity. A colour could be reconsidered across the work as a relationship, rather than treated only as a mark already laid down. Cohen wrote that the tools had given him a new way of thinking about colour. He also recognised that he was now working inside the program’s space, where previously its drawings had been brought out onto a physical surface for his contribution.[5]

Cohen’s example leaves creative work with a useful question. When we build a system around a craft, what becomes easier to examine, vary or understand? AARON supplied him with images, but the sustained conversation was about how images came to be. Teaching a machine to act became another way for an artist to keep learning.

Report an error or suggest a correction