Algorithm’s party tricks include turning pictures of horses into zebras

Researchers in an artificial intelligence lab in California have developed an algorithm capable of clever image manipulations, including turning images of a horse into a zebra.

The details of the work led by Jun-Yan Zhu and Taesung Park, from the University of California Berkeley, are laid out in a image-rich new paper.

It’s pretty technical, but the horse-themed images provided among the many examples given by the research team are very cool indeed.

Their work is in the field of image-to-image translation.

In their paper, they describe their development of a system that can special characteristics of one image collection and figure out how these characteristics could be translated into the other image collection.

However, crucially, their model is able to do it all in the absence of any paired training examples.

Some transformation efforts produced by the algorithm.
Some transformation efforts produced by the algorithm.

Years of research in computer vision, image processing, and graphics have produced powerful translation systems in the supervised setting, where example image pairs are available. This latest model takes it all to a new level.

The researchers from the UC Berkeley AI Research laboratory have even made their code available.

The algorithm is not only capable of turning horses to zebras and vice versa, but winter to summer, and  can even convert paintings to photographs.

It also did a pretty good job of transforming images back, with a couple of examples given below.

Two examples of the algorithm converting images back. The first one shows a photograph being converted to a Cezanne-style painting and then back to an image. The second shows a horse being converted to a zebra and then back again.
Two examples of the algorithm converting images back. The first one shows a photograph being converted to a Cezanne-style painting and then back to an image. The second shows a horse being converted to a zebra and then back again.

“Although our method can achieve compelling results in many cases,” the researchers reported, “the results are far from uniformly positive.

“On translation tasks that involve color and texture changes … the method often succeeds.”

However, there can be failures. They explored tasks that required geometric changes with little success.

“Handling more varied and extreme transformations, especially geometric changes, is an important problem for future work.”

Another transformation, with the input image at left.
Another transformation, with the input image at left.

They noted a lingering gap between the results achievable with paired training data and those achieved by their unpaired method.

“In some cases, this gap may be very hard – or even impossible,– to close,” the study team said.

“Integrating weak or semi-supervised data may lead to substantially more powerful translators, still at a fraction of the annotation cost of the fully-supervised systems.”

The paper is titled “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks.”

It can be read here.

Leave a Reply

Your email address will not be published. Required fields are marked *

Send this to a friend