In 9th-century Baghdad, Irak, three scholars known as the Banū Mūsā brothers wrote a treatise on mechanical devices and automata entitled The Book of Ingenious Devices. The book, published in 850 AD, described around 100 different devices and how to use them, including the first known programmable musical instrument.
The instrument was a hydraulic organ that worked using a system of interchangeable cylinders. The Banū Mūsā brothers called it "the instrument that plays alone". It used a pipe with holes—similar to a flute—, allowed the rhythm of the music to be changed, and could play different melodies:
“We want to explain how to build an instrument that plays alone continuously the melody we want, sometimes with a slow rhythm and sometimes with a fast rhythm, and we can also change from one melody to another if we wish.”
The Book of Ingenious Devices, Banū Mūsā brothers.
On the surface of the cylinder, there were wooden or metal pins of different lengths which, depending on their placement, opened or closed the valves of the pipe. The air flowing through the pipe, which was what created the sound, was generated by compensating water cisterns.
Although cylinders with pins on the surface were the most widespread method of producing music mechanically until the second half of the 19th century, this is much earlier, and usually, the moving force wasn’t water.
It seems that the organ was actually built, not just a theory, like other mechanical instruments described in ancient manuscripts. Sadly, the only known copy of The Book of Ingenious Devices doesn’t include the diagrams referred to in the text.
What we do have are detailed technical descriptions. If you’re interested you can find them in the book The Organ of the Ancients from Eastern Sources (1931) by Henry George Farmer. Also, a reconstruction was built for an exhibition that took place at the ZKM a few years ago. I assume by the image that the reconstruction worked with a small motor, not with water, but I’m not sure.
As I said, the organ played music in a loop, one or more tunes depending on the size of the cylinder used. One of the methods mentioned for moving the whole mechanism was to place a donkey, or a mule, going round and round—as in the old mills for grinding grain—but the text suggests that water is a better method than any kind of animal or just the wind because it allows a uniform continuous reproduction.
Farmer's book also includes descriptions of the notes and scales generated, in case you’re more interested in the musical side.
The only translation I’ve found of The Book of Ingenious Devices is—I guess—partial. Although it includes descriptions of some hydraulic machines based on the same principles as the organ, none of them is a musical instrument.
These kinds of automata are so wonderful that at the time they were seen as magic. In fact, many of the Banū Mūsā devices were a bit like magic tricks, such as a vessel that sometimes poured out wine and sometimes water—without refilling it. The funny thing is that one of the most famous automata in the world was indeed a trick, or to be more precise, a hoax.
The Mechanical Turk
The Mechanical Turk was a chess player automaton built by Wolfgang von Kempelen to impress the Empress Maria Theresa of Austria. Von Kempelen was a Hungarian author and inventor who unveiled his automata in 1770. It was a huge success and many people wanted to see the machine, but Von Kempelen was afraid because the machine didn’t play chess, who played was an expert chess player hidden inside of it.
The mechanism of the hoax was complex. It involved, among other things, a series of levers to move the arm of the doll and to open and close its hand. But it was a hoax anyway, so Von Kempelen didn’t want to publicise it too much and he only showed it a couple of times. Later, in 1781, Emperor Joseph II from Austria asked for the Turk to show it to the Grand Duke Paul of Russia and Von Kempelen couldn’t say no to an Emperor.
The Grand Duke was impressed and asked Von Kempeled to tour through Europe. At that point, Von Kempeled felt more confident, or perhaps more ambitious or needier, and he started a tour. He went to France, the United Kingdom, Germany, Nederlands, etc. Some people suspected that the Turk was a hoax, but nobody could really prove it.
The Turk kept performing around the world even after Von Kempeled’s death. It was bought by a musician who showed it to Napoleon I of France and many other European aristocrats.
In 1826, the Turk even travelled to the USA and then Canada. Edgard Allan Poe saw it and wrote about it:
“There have been many attempts at solving the mystery of the Automaton. The most general opinion in relation to it, an opinion too not unfrequently adopted by men who should have known better, was, as we have before said, that no immediate human agency was employed — in other words, that the machine was purely a machine and nothing else. Many, however maintained that the exhibiter himself regulated the movements of the figure by mechanical means operating through the feet of the box. Others again, spoke confidently of a magnet.”
Maelzel's Chess-Player, 1836, Edgard Allan Poe.
The machine wasn’t always operated by the same person, so as time went on, it was more and more dangerous that one of the chess players inside revealed the secret. A lot of writers and journalists tried to unveil it, and some of them had theories that were somewhat true, but no one really knew how it worked. The real mechanics of the machine were revealed many years later.
Anyway, I'm not so interested in the exact details of how it worked—you can find them online. I'm much more concerned about what this automaton represents and why it gave its name to Amazon Mechanical Turk.
Amazon Mechanical Turk
Amazon Mechanical Turk, also known as MTurk, is a “crowdsourcing marketplace that makes it easier for individuals and businesses to outsource their processes and jobs to a distributed workforce who can perform these tasks virtually”. What this means is that it’s a website in which people are very poorly paid for playing chess hidden inside a Mechanical Turk. Sort of…
A company needs some work to be done and they don’t want to pay employees, so they hire another company, MTurk in this case, which divides the job into microtasks and assigns them to users who receive a small amount of money in exchange for doing those microtasks. Each of those microtasks has a price assigned, sometimes just $0,1 or even less.
According to a 2018 study, the best-paid workers in MTurk earn between $1 and $6 per hour. In many countries, that’s well below the minimum wage. The federal minimum wage in the USA, where Amazon is from, for covered nonexempt employees is $7.25 per hour. As Jeanette Winterson writes:
“The Turk made plenty of money in rake-offs—much as Amazon does now on its own MTurk platform, built to cash in on hard-pressed workers biding for jobs. Life is on repeat.”
12 Bytes: How artificial intelligence will change the way we live and love, 2021, Jeanette Winterson.
I don’t know who came up with the name Amazon Mechanical Turk, but it’s obvious where it comes from and it’s a bit on the nose and insulting. Come on, it was literally a hoax machine which hid the human worker. Most microtasks available in MTurk are related to AI, which works like the Mechanical Turk. AI isn’t intelligent, it needs a human. To be more precise, many humans. That’s what some people call “ghost work”.
“Artificial intelligence depends on human labour to conduct tasks such as data cleaning, coding, and classifying content. This on-demand work is offered and performed online, paid by the task, on platforms like Amazon Mechanical Turk. Conceptualized as ‘ghost work’, this rapidly growing, platform-based work is largely unseen: workers are unable to speak with managers, do not get feedback, and lack labour protections.”
There was a time, a few years ago, when I was intrigued about AI/ML training, especially for linguistics. I worked as a translator for many years and I’m interested in linguistics, so I was really curious. I’ve never worked for MTurk, but I used similar websites, where I worked—always through third parties—for Google Quick Answers, Google Assistant, and similar technologies.
I worked as a freelancer, and there weren’t any feedback, managers or labour protections. I received a user and a password to use a proprietary online tool in which you could see brief instructions and had access to a batch of files. That’s all.
Maybe you’ve seen the film Kimi (2022) by Steven Soderbergh. I did exactly the job that Zöe Kravitz does in the film, creating training phrases and reviewing recordings by the users. The problem is that the character from the film makes tons of money, judging by where she lives, and that’s not the reality of that kind of ghost work. The film is good, but that detail couldn’t be further from the truth.
Around 2014, when I started, the salary was ok, it was not high, but it wasn’t either insulting. It was higher than the minimum wage in my country, which is not very high—€7.82 per hour—, but there’re many countries in which it’s much worse. However, as AI and ML became more usual, the pay got lower and lower, until eventually, they started paying with Amazon gift cards. I quit long before that, but I'm sure there’re desperate people out there who accept those awful conditions.
Another problem is those AIs are being trained by who knows who. They’re not employees, just random people who signed up on a website. Maybe your 60-years-old sexist uncle is training the next language or image model.
A well-known case of this problem is ImageNet, which at some point removed more than 600,000 images from its database because they were exposed as racist. ImageNet is a database that has been used for many machine learning projects. The images were categorized by MTurk workers and a lot of the descriptions were racist, sexist, etc.
All of us have biases, but it’s not the same to train a model using a diverse pull of employees that a random group of mostly white and poorly paid ghost workers.
I started this newsletter writing that now we’re the automata, but we’re ghosts, and being a ghost is even worse. Invisible dead entities. I wish we could at least scare the master of the castle, but we don't even have that power.
What do you see, YOLO9000?
What do you see, YOLO9000? is a short experimental film that reflects on how trained models label images. The film is funny but also disturbing because it shows a lot of the usual issues and biases. It’s in English, the only text in Spanish is the brief intertitle at the beginning:
“YOLO9000 is an object recognition neuronal network. It is one of the many artificial vision tools designed for automatic image annotation.”
I think that their reflection speaks for itself. The film is by Estampa, a group of filmmakers, programmers, and researchers based in Barcelona who works with audiovisual and digital environments.
AI Deviations
We often think about AI in terms of image and text, but AI can be used for many different things. As this week’s newsletter started with a music automaton, I wanted to finish with an example of how some sound artists are using AI.
You may already know Yasuano Tone, he’s 87 years old, so he has a long career behind him. He’s a multi-disciplinary artist, but he’s mainly known for his sound pieces. Around 1984, he started manipulating CDs to generate unique glitched sounds. Later he started working with other digital media, such as MP3 files. In 2017, he published an album generated using AI.
Tone explains that around 2002 he had this idea of applying a neural network to create a sound piece. At that time, he didn’t have the means to do it, but in 2015 he got a grant that made it possible.
“I had talked about the idea with Prof. Tony Myatt at Surrey University, UK and he developed the software for the piece with a team included Dr. Paul Modler. At the lab in the University a series of my performances of my MP3 Deviation were captured and used to train Kohonen Neural Networks to develop artificial intelligences that simulate my performances.”
Yasunao Tone
The album is the recording of a performance that took place in 2016.
I wrote this last part of the newsletter while cooking some vegetables in the oven. It’s artichoke season. Autumn is coming!
As always, if you have something to say you can find me here and on Twitter. You can also support this newsletter with money if you feel like it.
And you can also share it, of course, it helps a lot and it’s free.
You can find most of my audiovisual work on my website: null66913.
Wherever, whatever, have a nice day :)