Optical sound—or sound-on-film—is one of my main obsessions. Well, not optical sound per se, but films that use it to read visual data as audio data. The technique, or the theory behind it, is simple. For several decades, the soundtrack of motion pictures was recorded on the same filmstrip that the images, using optical means. That soundtrack can be seen, at first sight. It’s like the waveform that you see nowadays on most audio software, but printed on a filmstrip instead of visualised on a computer.
Here you have an old video in which they explain how it works. Some of the explanations are a bit technical but it’s easy to understand the basics, even if you don’t know much about how sound and optics work.
This was the main system for film soundtracks from the 1920s to the 1950s, when the magnetic tape started becoming more popular. The fascinating thing about optical sound is that many artists and engineers had this strange idea that, as the sound was a “drawing” on the filmstrip, they could draw synthetic sounds instead of recording sounds from real life. Some of them called it “ornamental sound”, others “graphical sound” or “drawn sound”.
The pioneers
Oskar Fischinger
The most well-known of these artists is Oskar Fischinger, who was a painter and filmmaker from Germany. In 1933, the magazine Radio Review and Television News published a text explaining how he created music harmonies from geometric patterns.
The correspondence between image and sound is not random, after a bit of research, he learnt how to create specific notes and different kinds of sounds. He could generate delicate or abrupt sounds, for example.
“Flat waves will give soft notes, sounding as though from a distance, whereas those drawn at right angles correspond to sounds of normal strength and steep waves to sounds of excessive intensity.”
Radio Review and Television News, Jan-Feb 1933
Fischinger didn’t draw directly on the filmstrip, he drew on paper and photographed the drawings. This was the standard practice because a filmstrip is small and it’s easier to draw on a roll of paper.
Arseny Avraamov
At the same time in Russia, the composer Arseny Avraamov developed the same technique. He hand-drawn geometrical patterns that represented sound shapes and photographed them on filmstrips.
In 1930, he founded the Multzvuk group inside the Mosfilm Productions Company to research these techniques. They created music for several experimental films, but all of them were destroyed at some point between 1936 and 1938 by Avraamov’s sons. It was an accident, they were making rockets using celluloid filmstrips, which are highly flammable, and all the films burnt.
Rudolph Pfenninger
There was at least another engineer and artist who developed the same technique at the same time, the German Rudolph Pfenninger. This might sound suspicious but we have to remember that before the Internet it was usual that several people arrived at similar conclusions at the same time without knowing each other.
Pfenninger had both engineering and artistic knowledge. When he met the American producer and director Louis Seel in 1921, Seel hired him to draw animations and titles for his films. A bit later, Pfenninger found a job at EMELKA, one of the biggest film studios in Germany. There, he started researching radio and audio techniques. That’s how he developed an interest in synthetic sound. A curious detail is that he made experimental films and, as he didn’t have much money, he was interested in generating sound without having to pay musicians.
He identified the graphical shape of each note with the help of an oscilloscope and started drawing those shapes on paper. Then, like Fischinger and Avraamov, he photographed those drawings on filmstrips. Some researchers think that his film Pitsch und Patsch (1932) was the first one to use synthetic sound, but it’s hard to know for sure.
Most film critics thought that his ideas were amazing, but they also thought that the resulting sounds were disturbing and soulless. In other words, for them it was interesting from a technical point of view, but not from a musical or artistic one. The general public was more open and those EMELKA experiments were shown in most European countries. The real reason why this experimental genre disappeared from the mainstream theatres was not the public, was that the nazis hated it. They thought that the films were degenerate and they did all in their power to stop them.
Most of these films were destroyed, lost, or are not available online, but you can have an idea of the kind of sounds and animations that they were creating by watching this short documentary.
Other contemporary filmmakers
Some years later, some experimental filmmakers started using again these techniques. The most well-known is Norman Mclaren, who began experimenting with hand-drawn sounds in the late 1930s. McLaren made many films using this technique from the late 30s until the 70s. Unlike his predecessors, he usually drawed directly on the filmstrip, not on paper. He made a great film explaining how he created his optical soundtracks. It’s much easier to understand it seeing him working.
There’re many contemporary filmmakers that have used or use this technique, some of my favourites are Lis Rhodes, Guy Sherwin, and Bruce McClure.
Rhodes has a film—Dresden Dynamo (1971)—in which the images are in the whole filmstrip—both on the space for the image and on the space for the soundtrack—, so what you see is exactly the same that what you hear.
In films such as Railings (1977), Sherwin works with live footage, not with drawings, but the technique is the same. In this image is not so easy to see the soundtrack because the frames are in black-and-white, but the sound is on the top part.
McClure creates visual performances, not films. He uses 16mm projectors and processes the sound live using optical sound synthesis. He says that his work is a bit like a voluntary torture based on noise and intermittent images.
Many younger artists who work with subjects related to synesthesia and the links between eyes and ears also use the optical sound technique nowadays. One of them is Mariska de Groot, who creates installations and performances based on analog light-to-sound techniques.
Here I’m talking mainly about film and visual arts, but optical sound has also been used for musical instruments: the Variophone, the ANS, the Oramics, etc. But that’s another story that I’ll revisit in a future newsletter because I like to keep this simple and short. I’m trying to write texts that anyone can understand and don’t take much of your time.
Nowadays, working with optical sound is not so easy. You need film, projectors, a laboratory, etc. It’s not really that expensive, because you can find cheap old 16mm projectors, but you need much more money and space than if you use a computer. That’s the main reason why I haven’t worked with film for a long time, but I wanted to listen to images anyway, so…
Digital image-to-sound techniques
I spent several years searching for a way to replicate optical sound using digital media. The good news is that you can do it. You just have to use RAW files, which can be opened both in audio software such as Audacity and image editors such as GIMP.
I’ve been working using this technique for films since 2011. Before that, I used to generate still images from sound files. It was also fascinating, but I’m a filmmaker, I’m not so interested in still images. My first experiments were focused on generating sound from JPG frames, later I began also to generate images from audio files. The first tests were not that interesting, but after a decade I have some films that I think are quite good.
Image-to-sound
Engram (Optical Sound #001) is part of a series entitled This Is Not Cinema that recreates traditional experimental cinema aesthetics using lo-fi digital video tools. In this case, I was obviously inspired by optical sound experimental films.
I made this using live footage. It’s abstract, but what you see is a lightbulb turning on and off recorded with an iPod Touch, using an app called 8mm that recreates 8mm film aesthetics. The sound was generated directly from the frames, exporting them to JPG and then converting them first to RAW and then to AIFF. It's not optical sound, of course, but it's based on similar ideas. Curiously, the resulting sounds are similar to those from optical sound experimental films.
RGB Colour Model is an animation based on the additive synthesis RGB (red-green-blue) colour model—the primary colours of light. The soundtrack, as in the previous example, are the film frames themselves saved as audio files. What you see is exactly the same that what you hear.
This game of flickering shapes and noises generates entoptic phenomena—visual effects whose source is within the human eye—and auditory hallucinations.
Sound-to-image
Computer Music Studies is a series of audiovisual works based on digital feedback. The sound was generated by Mikel R. Nieto in an autonomous way using different configurations of the same pattern. The audio output returns to the audio input generating an internal feedback that is nonlinear and difficult to control.
The video, created by me by hand, converting each frame, one by one, after splitting the soundtrack into fragments of around 41.67 milliseconds, shows the sound.
You can find more of my films, many of them using these same techniques, on my website null66913. If you want to know more details about how I made them you can ask, I’m not secretive about my work. Also, I wrote a related article a few months ago, there you can find a simple tutorial to generate images from sound, and vice-versa.
If you want to know more specifics about optical sound, I also recommend these two texts:
Tones from out of Nowhere: Rudolf Pfenninger and the Archaeology of Synthetic Sound, Thomas Y. Levin, 2003.
Sound in Z Experiments in Sound and Electronic Music in Early 20th Century Russia, Andrey Smirnov, 2017.
So much for now. Are you reading this for the first time? You can subscribe, it’s free!
If you’re already a subscriber and would like me to continue writing this newsletter, you can support me on ko-fi, by buying something in my shop or simply by sharing this.
Have a nice week!