In what is called “neural karaoke,” this project from the University of Toronto programs computers to take pictures and compose music about them–and it involves a multitude of steps. Researchers Raquel Urtasun, an associate professor in computer science, Sanja Fidler, an assistant professor in computer science, and PhD student Hang Chu, started by giving the computer 100 hours of online music to analyze. Once it could generate its own music, they moved to teaching it to analyze dance moves in order to create choreography. In the final step, the program was given a collection of pictures and captions, analyzing them to see how words and visual patterns are linked. Then, given a fresh image–say of a Christmas tree with presents wrapped under it–the program creates and sings a song. If the program can’t say the words it has chosen, it substitutes an “Ooooh” sound–which may explain some current music, now that I think of it..
The current composition, though charming in its own way, is not going to be in anyone’s top ten, but the program’s creators have grand ambitions, including seeing this as a household alternative or addition to Pandora. Fidler suggests that “Instead of buying a karaoke machine with certain tracks on it, you can create your own karaoke at home by throwing in some interesting photos and inviting the machine to generate music for you…I think it has endless possibilities.”
Wishing you a very Robotic Christmas with lots, and lots, and lots of flowers!
via The Guardian