Lighting, Layout, and Pose Tools for Artistic Drawings Using Machine Learning

Abstract

Machine learning algorithms that work on high-resolution images fast developed since the early 2010s. Much research solves problems from a photograph, such as object detection, segmentation, human face manipulation, and various image-to-image translation tasks. Moreover, much machine learning research can inspire each other since many problems are in the image understanding domain, though the applications are different. Benefiting from the fast-growing machine learning research, we can look into more intelligent solutions for computer-aided drawing. We explore three application-oriented research using machine learning to help humans manipulate artistic drawings in lighting, layout, and pose tools. The first work, ”Learning to Shadow hand-drawn Sketches,” presents a fully automatic system to generate self-shadow from the 2D line drawing. The generated shadow can be a starting point or hint for the artist to draw a shadow on a 2D line drawing. We used conditional generative adversarial networks (GAN) and contributed some novel modules and loss functions for processing the sparse data 2D line drawings. In the second work, we explore a pipeline to learn the aesthetic art composition principles from art masterpieces into a machine learning model. Then use the trained model to help suggest beautiful art composition plans. Finally, in the third work, we develop a pose estimation system for children's art to detect the shoulders, hips, et cetera. This pose estimation system can help to animate children's drawings in the wild automatically.