The Process
Motivation
As a computer science student who also majors in art history, I have long been fascinated by the application of neural networks in image generation. After researching into GANs, WGANs, CycleGAN, pix2pix and more other image generation neural nets, I noticed that few existing networks engages with abstract images. There are a few theoretical reasons behind this reality, most due to the inability to label abstract art and the diversity of abstract visual sensibilities.
Instead of answering questions surrounding the specificities of abstraction in deep learning, this project uses deep learning as a tool of art making. I intend to takes advantage of the universality of neural networks to learn and create abstract images. Not only is this project significant in its contribution towards computer science as a novel application of neural networks in visual production, but this project also challenges our art historical understanding of abstraction and the human genius.
background figure: Franz Kline, Painting No.7, 1952
I. Topic Selection
An “American Style”
For this project, I decided to work with primarily black and white works by Abstract Expressionist artists Franz Kline (left) and Robert Motherwell (right). Personally, I have engaged with Kline and Motherwell repeatedly in my art historical career and am amazed by their artistic talent. Technically, the two share a simple and distinctive style that I believe, intuitively, are easier for neural networks to learn.
figures below: Franz Kline, Number 2, 1954. & Robert Motherwell, Elegy to the Spanish Republic No. 70, 1961.
II. Data Collection and Data Processing
One of the biggest challenges of this project is the scarcity and decentralization of data. There are very few contemporary art databases out there, unlike art from previous times. Artworks by American Abstract Expressionist painters are collected in tens if not hundreds of galleries and museums across the world. For this project, I manually searched through all digital catalogs of major museums and galleries, and used a script to download images that suffice certain criteria.
I edited these images into certain sizes, colors, and ratios that make them easier to learn for neural networks. In the end, we have a database of 500+ artworks from these artists - the largest existing database on these artists in the public domain.
A Sample of the Database
III. Network Selection and Training
GANs
The first network I used is a simple GANs network. This network is fairly standard, but I included features such as LeakyReLU and Adam optimizer to ensure maximum generalization. After learning for 2000 epochs, however, the network attained very little. The final output is low in resolution and high in noise. The network generalized few features, and is clearly mode collapsing.
figure: output from GANs after 2000 epochs
2. DCGANs (Deep Convolutional GANs)
The second network I used is DCGANs (https://arxiv.org/abs/1511.06434). I modified an existing PyTorch implementation of DCGANs (https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) to provide more convolutional features. The result of this network is much better. Upon visual analysis, the network has learned to generalize brush strokes thanks to convolutional mechanisms. Many of these images are very unique and interesting. However, the output still yields low resolution (50*50) and high noise.
figure: animation of output from DCGANs after 5, 50, 100, 150, 200, 250, 300, 400 epochs
Select works made by DCGANs*
* the images are digitally enhanced (noise reduction, contrast increase, sharpened, etc.) for artistic purposes
3. ProGAN (Progressive growing of GANs)
In order to increase resolution and generate more details, I looked to ProGAN (https://arxiv.org/abs/1710.10196). I used a TensorFlow implementation on GitHub (https://github.com/tkarras/progressive_growing_of_gans) and adjusted the network for the purpose of this project. The network produces high-definition images (128*128 since the input is 128*128) with significantly less noise. The network can also reproduce the original data with extreme precision.
The downsides, however, are two-fold. First, the network is extremely computationally intensive. It takes around 2 weeks for the network to train on a single-GPU machine with one NVIDIA Tesla P100. The final images took 2 days of training after I gained access to a state-of-the-art 8-GPU machine. Second, the network requires a large database. Due to the nature of progressive growing, the network can easily memorize the inputs since the network learns for around 30,000 epochs over this small dataset. The output, therefore, suffers from mode collapse and overfitting.
figure: animation of output from PROGANs at each resolution
Select Works by ProGAN
IV. Further Research
Large dataset on contemporary art
Dataset on modern and ancient art is fairly well made thanks to efforts from WikiArt and researchers from CycleGAN. It is important to create well-constructed datasets on contemporary art so that we can engage with contemporary art through artificial intelligence.
Generative networks for small datasets
In Franz Kline’s case, his tragically short-lived career leaves us few works to learn. It is important, therefore, to design networks that can learn without overfitting on a relatively small dataset. This effort will be helpful since many visual data we are working with may not come in abundance.
Qualitative studies on generating abstract images
One of my original fascinations is to see if neural networks work with abstract images as well as they do with representative images. Through this project, I have noticed that the generation of abstract images is possible yet challenging. It is important to conduct quantitative studies to understand what the networks are learning when they are working with abstract images.
