What follows could lead to addictive behavior. Proceed at your own risk!
Odds are that your phone, tablet and laptop are loaded with many hundreds of photos and tons of data that can be transformed into exotic melodies unlike anything you’ve ever heard. In this review, we will examine two programs that transform images into sound. We will then look at musicalgorithms, a program that transforms strings of data into sequences of proportional musical notes.
Extracting Melodies from Photos
While the digital photos stored in your devices are the result of amazing solid state and optical technology, in one respect they are just as primitive as the very first photos made more than 150 years ago.
The missing element in still photos is sound, but change is in the air. Thanks to the two clever programs reviewed here, you can transform any or all of your photos into completely unexpected melodies. These compositions can range from a basic series of multifrequency notes to complex sequences of various durations complete with adjustable frequencies, variable harmonics and both in- and out-fading. By tinkering with the adjustable parameters, unique and even memorable melodies can be extracted from carefully selected photos. You can add these tunes to your PowerPoints, blogs and social sites.
How these programs do what they do is not intuitively obvious. What can be deduced by experimenting with them is that they scan the vertical columns of pixels in a photo or drawing from left to right. Each column of pixels is then digitized and transformed into a representative audio tone. As the program sweeps across the adjacent pixel columns, the resultant tones are merged into a melody. Images filled with contrasting shapes and patterns produce the most complex melodies.
Getting Started with SonicPhoto
My favorite photo-to-sound program is SonicPhoto, the free demo version of which will provide hours of fun for you and your friends. The demo program permits compositions lasting up to ten seconds that can be saved as WAV files you can post online or send via email. Two purchased versions of the program are also available. The Silver edition ($29) permits compositions up to 45 seconds long and the Gold edition ($69) has no limit.
The best way to evaluate SonicPhoto is to load the free demo version and listen to some of the 78 preset samples of public domain photos that are accessed by clicking on the window at upper left in the screenshot above labeled Select a preset! Then click the green Create or Play Sound button to hear the photo.
One of my favorites is “Space Mosquitoes,” the fireworks display (1) shown in the screenshot. This sample preset yields a complex melody that has to be heard to be fully appreciated. The screenshot shows the SonicPhoto controls adjacent to and below the fireworks image. As in all the samples, most of the controls are easily adjustable sliders. So after playing “Space Mosquitoes” a few times, it’s time to experiment!
First, click Rotate 90° at the top center of the page and then click the green Play button. This will give you a good idea of how the orientation of a photo affects the pitch of the tones that are produced as the program scans the image left to right.
After unchecking Rotate 90°, try tinkering with the Fade in and Fade out sliders to see how they alter the dramatic effect of the melody. When you’re happy with your selection, adjust the sliders for the frequency range, the duration of the melody (up to 10 seconds) and the various other options. If your version of the melody goes badly, the original version in SonicPhoto is unaffected. Just reload the sample and start over.
Tinkering with the SonicPhoto samples will give you a good idea of how best to select your own photos for conversion to melodies. Photos with lots of distinctive, high contrast elements will provide the widest variety of tones. Try stair steps, buildings poking up into the sky, rows of street lights, illuminated Christmas trees, stars at night, trees that have shed their leaves and so forth.
You can better understand how SonicPhoto works by loading geometric shapes into the program. Feeding the program a dark triangle will give a tone having a frequency or pitch that gradually rises and then gradually falls.
A row of horizontal bars gives a uniform tone. But rotate the bars 90 degrees, and the program produces a sequence of chirps or tones, one for each bar. This confirms that the program is scanning across the photo from left to right.
Bars may look boring, but single bars and patterns of bars in the real world can produce delightful sounds. For example, this photo I made of a brilliant green laser lidar beam extending well into the stratosphere over Hawaii’s Mauna Loa Observatory provides a pleasing chirp when scanned by SonicPhoto. But don’t take my word for it. Here’s both the image and the sound it produces.
Moving Up to PhotoSounder
SonicPhoto developer Daniel White acknowledges on his website that his program was “Inspired by the existing PhotoSounder program from Michel Rouzic.” While both programs produce exotic results, Photosounder includes a set of advanced features that must be seen and heard to be fully appreciated. As Rouzic explains on his website: “Photosounder truly allows you to transform any sound as an image and to create any possible sound from an image. It is the ultimate bridge between the graphical world and the audio world, bringing the full power of image editing to the service of creating and transforming sounds.”
Rouzic is not exaggerating, and you can find out why in just three mesmerizing minutes. Just click on “The Sound of Fractals and Photographs” YouTube link at the upper right of the Photosounder home page. I feel safe in promising that you will not be disappointed. In fact, as these words are typed, the amazing audio from that link is playing in the background.
As shown in the YouTube video samples and the screenshot below, an especially useful visual feature of Photosounder is a vertical index line or cursor that moves across the screen from left to right as the pixels along it are transformed into tones of varying pitches. Watching the cursor provides intuitive clues that will help you select photos best suited for conversion to sound. For example, faces alone might sound rather bland, but clothing with polka dots, stripes or complex patterns will solve that lapse.
The many controls provided by Photosounder can be seen in the screenshot above that shows the same “Space Mosquitoes” image used in the SonicPhoto discussion above. Both programs produce similar audio effects while Photosounder provides the left-to-right moving cursor line and a color-coded intensity scale across the top.
Virtual knobs along the bottom of the screen control volume, gamma, intensity, frequency range and other parameters. I found these knobs more difficult to adjust than the sliders in SonicPhoto. You can select a minimum frequency as low as 0 Hz and an upper frequency as high as 26,000 Hz, which might annoy any nearby dogs. The frequency range you select is indicated along the left side of the screen in Hertz and along the right side in musical pitches. Centered within these controls is a large button with an arrow that’s clicked to launch the audio conversion process.
As with SonicPhoto, some key features of Photosounder do not function in the demo version. You cannot save your compositions, and a brief silence interval is inserted every 12 seconds. But trust me, you can have at least as much fun experimenting with the demo Photosounder as with the demo SonicPhoto. The full version of the program is $79. See the website for details.
Transform Data to Melodies with musicalgorithms
Transforming strings of data into melodies might seem primitive when compared to doing likewise with photos and drawings. But there’s a distinctive role to be played by musicalgorithms, for just as a sculptor reveals a figure encapsulated within solid rock, this program extracts the melodies hidden within your bank account, sports statistics, stock quotations, weather data, tables of numbers and even mathematical constants such as pi.
SonicPhoto and Photosounder can extract an exotic and even entertaining melody from a photo of the annual growth rings of a tree. Musicalgorithms (pronounced music algorithms) is more literal, for it can create its own unique sequence of tones from a list of the differing widths of all the rings. The result in this case is a tonal sequence that can be adjusted so that narrow rings signifying drought years have a low pitch, while thick rings produced during rainy years have a high pitch.
In other words, the pitch of each tone produced by musicalgorithms is proportional to the magnitude of the number from which the tone is derived. Therefore, the tonal sequences from a string of numbers is an organized, audio representation of the data. While the pitch of tones from a tree ring record jump around, those in a sequence extracted from a daily or seasonal temperature record smoothly rise and fall when the temperature increases and decreases. This proportional conversion of numbers into sound provides a blind person with an entirely new way to grasp changes in data.
Musicalgorithms was developed under the leadership of Jonathan Middleton, Professor of Composition at Eastern Washington University, where he teaches courses in composition, orchestration, and computer music. Prof. Middleton earned a Doctor of Musical Arts from Columbia University, where he was an Andrew W. Mellon Fellow at the School of the Arts. The development team for the version of musicalgorithms described here included Anthony Della Maggiora, Kristi Marks, Michelet Chery, and Dylan Paulus. The project was originally sponsored by the Northwest Academic Computing Consortium.
How to Use musicalgorithms
You don’t need to download a program to use musicalgorithms. Just click here and begin tinkering with the options. An alternative site that requires Java is here. Before entering a custom string of data, try one of the embedded options to get a feel for what the program does. Let’s start with one that will give you an entirely new perspective on pi (π), the mathematical constant (3.14159…) for the circumference of any circle divided by its diameter. At the Welcome page, note the options listed across the top of the page and proceed as follows:
Pitch Input: Click here to select one of the embedded demos or enter your own data (Custom). For pi, click the window labeled Input set and click Pi. In the Note Count window enter a number up to 2,000.
Duration Input: Click this option at the top of the screen and again select Pi.
Pitch Mapping: This step at the top of the screen normalizes the pitch value for each digit of pi to a proportional scale. Selecting a range of 1-88 is equivalent to selecting an 88-key piano.
Duration Mapping: This step sets the duration of each tone in the sequence. The default range of durations is 1 to 6. For initial experiments, change this from 1 to 2 to assign the same duration to each tone.
Scale Options: If you have musical experience, select anything you like. For novices, select Chromatic in the first window and C in the second window.
Play: You’re ready to listen to pi! Click Play at upper right, and a piano keyboard with 88 keys will appear under color-coded buttons labeled Play, Pause and Stop. Directly over the Play button is an Instrument window. Click this and select Vibraphone (sounds better than Piano). Then click the green Play button to hear the audio rendition of pi. Keys on the keyboard will turn red in response to each note being played, and progress will be shown by a slider at left. You can manually adjust a slider at right to speed up or slow down the melody of sounds.
This audio rendition of pi covers only the basics of musicalgorithms, so be sure to experiment with all the settings to find those that you prefer. You’ll then be ready to import your own data.
Entering Your Data into musicalgorithms
The only requirement for entering custom data into musicalgorithms is that the numbers must be separated by commas (CSV or comma-separated values). Get started by returning to the Pitch Input option at the top of the screen. Then click Input Set and select Custom. Next, clear the “0” and enter this string of comma-separated values into the Input window: 1,2,3,4,5,6,7,8,9,10. Then enter “10” into the Note Count window.
Under Duration Value, select Custom and ignore the 0 in the window. Skip back to Pitch Input and make sure the Note Count window is still set to 10 (it may default to 1 after you make changes in subsequent windows as we did under Duration Value).
Now you’re ready to hear your string of 10 notes. Click Play at upper right to see the keyboard page. Select Vibraphone in the Instrument window. Slide the Tempo pointer all the way to the right and click Start. After you play with this initial trial, you’ll be ready to load strings of real data.
It’s likely that the numbers you want to enter are not separated by the required commas. You can add the commas with the help of a spreadsheet like Microsoft Excel or LibreOffice Calc, a free and very capable program. First, paste or manually enter the numbers into successive cells along a single row of the spreadsheet. If the numbers are in a vertical column, transpose the column into a horizontal row by highlighting and copying them. Click on the cell where you want the row to begin. Then click Paste Special, Transpose and OK. The vertical column will be rotated into position along the row you selected.
Next, save the row of numbers in a CSV file (save as “CSV” in Excel or “Text CSV” in LibreOffice Calc). Then open the CSV file using Notepad in Windows or TextEdit in Mac OS. (When using a Mac, you may need to select the “Plain text” option). Your comma separated numbers will now appear in one or more rows. Copy the numbers and paste them into the musicalgorithms Custom option under Pitch Input. You can now adjust the various options and then play your number sequence. You can even save the composition as a MIDI file that you can further manipulate.
Finding Data for musicAlgorithms
It’s easy to find photos, drawing and art for SonicPhoto and Photosounder, but where do you find number strings for musicalgorithms? Fortunately the web is a gold mine for number strings. The easiest to find are time series of data that show both routine and unexpected fluctuations when plotted on a chart. Good sources are daily time series of the price of a stock, your bank balance, the low or high temperature of your location or anywhere else, the thickness of the ozone layer, the intensity of the sun’s ultraviolet rays, the traffic count at an intersection, the price of a barrel of oil or a gallon of gasoline, and ad infinitum.
Some sample musicalgorithms
I’ve described in my column in MAKE magazine several examples of tone sequences from data I’ve collected that are ordinarily plotted as lines or bars on graphs (“Making Synthesized Music from your Data,” October 2012, pp. 144-146). Here are some YouTube videos of those examples:
This video shows a sequence of 170 360-degree fisheye photos of the noon sky from January to December of 2010. The photos were made from the field by the tiny Texas farmhouse that serves as my office. To the right of each sky photo is a musicalgorithms keyboard playing the tones that represent the intensity of the sun’s ultraviolet radiation measured next to the camera each day. The low UV levels during winter give tones having a low pitch, while the high UV levels during summer are indicated by high pitched tones. Haze, clouds and the ozone layer further modulate the pitch of the tones. The variation in pitch that represents the seasonal change in UV will provide a blind person with a new way of appreciating this natural cycle.
A typical Geiger counter measures around 11 counts per minute at the ground and several hundred counts per minute at altitudes of 35,000 feet or more. The increase in the count with altitude is due to cosmic rays, and you can both see and hear this for yourself during a musicalgorithms flight from San Antonio, Texas, to Zurich, Switzerland. The pitch of the tones is proportional to the altitude.
Many trees produce one growth ring each year. The light colored spring growth is early wood. The darker summer and fall growth is late wood. The overall width of the rings is closely related to precipitation, with thin rings indicating drought years and thick rings indicating wet years. In this video, the width of the early and late wood within each ring from 1977 to 2010 of a bois d’arc tree that was downed by a flood on my property was measured and then loaded into musicalgorithms. The notes for each ring are in this sequence: early wood/late wood/total width/pause.
By now it’s quite possible that most of the readers of this article have become so addicted to transforming their favorite photos into works of music that they have not made it this far. For those of you who have arrived here, there are many online resources for fans of synthetically-generated music. For starters, check some of the audio features available with programs like Audacity (free) and Mathematica (not free). While many other possibilities are online, the three programs reviewed here are the best I’ve found.
The fireworks photo in the SonicPhoto and Photosounder screenshots is “New Years in Village,” a royalty-free, public domain image by Jon Sullivan. It’s available here.
About the Author
Forrest Mims is best known among electronic musicians for developing the Atari Punk Console. His research has appeared in leading scientific journals, and he was named one of the “50 Best Brains in Science” by Discover Magazine. His science is featured here. Email him at [email protected].