top of page

Synchronizing Music And Visuals

​

Colour Music

What Is Colour Music ?

​

Colour music is often confused as synaesthesia and sometimes a form of intermedia.

Colour music refers to systems that converts music or sound into visual forms,

such as film, video or computer graphics.

​

Converting images to sound by drawn objects and figures on a film's soundtrack, in a technique known as drawn or graphical sound which is a sound recording created from images drawn directly onto film or paper that were then played back using a sound system.

There are several different techniques depending on the technology employed,

but all are a consequence of the sound-on-film technology and based on the creation of artificial optical polyphonic sound tracks on transparent film. Famous visual music artists include Jordan Belson,

Oskar Fischinger, Norman McLaren, John Whitney Sr., and Thomas Wilfred.

Who Started  Colour Music ?

    Visual music dates back as far as the 16th century.

Through an Italian painter Giuseppe Arcimboldi,

his study of the Pythagorean harmonic proportions of tones and semitones displayed the relationship between the musical scale and the brightness of colours. 

    Experimenting with white and gradually adding more black, he managed to render an octave in the twelve semitones, with the colours ranging from white to black, this grey scale painting would gradually darken the colour white, using black for indicating a rise in semitones, with white representing a deep note and black representing high ones.

​

    In 1704, while analysing the spectrum of light, Isaac Newton suggested a close link between the seven colours of the rainbow and the seven notes of the music scale.

Newton stated that an increase of the frequency of light in the colour spectrum from red to violet made a corresponding increase in the frequency of sound in the diatonic major scale.

Since Newton’s idea, other people have had different response to his link between colour and sound.

In 1743, a French mathematician by the name of Louis Bertrand Castel introduced the relationship between colour and notes.

This led to him inventing and creating the ocular harpsichord,

this musical instrument could transform sound into colour.

With each note in the scale representing a different colour,

for example when the C note was pressed, a small panel indicating the colour violet would appear above the instrument.

The mathematician later perfected his system,

proposing a range of twelve colours, which corresponded to the semitones.

    A number of instruments and responses were since based on Castels work, all with their own ideas on the relationship between

colour and sound.

    With many studies on the relationship between colour and sound over the years, physician Ernest Chladni, took a different approach to the study and looked at the relationship between sound and form. 

In 1987 he investigated the patterns produced by certain frequencies through vibration on flat plates.

    This was achieved by scattering fine sand evenly over a glass or metal plate and by gliding a violin bow against the plate to cause patterns through vibrations. The vibratory movement caused the powder to move from the antinodes to the nodal lines. Black lines represented the parts of the plate, which vibrated the most.

Chladni was able to produce sound, giving it a dynamic image; he discovered that the same sound would produce the same pattern each time.

 Hans Jenny a swiss doctor was influenced by Chladni`s work in cymatics, which is the study of sound and vibration made visible.

In 1967 she published the first volume of cymatics. The Study of Wave Phenomena documented several experiments performed by Jenny using sound frequencies on various materials including water, sand,

liquid plastic and iron filings.

    Many crystals are distorted by electric impulses and produce electric potentials when distorted. When a series of electric impulses are applied to the crystal, the resulting distortions will have the character of real vibrations. 

These crystals allowed for a whole range of experimental possibilities with the ability to display both frequency and amplitude. The oscillator is attached to the underside of the plate and when a frequency is outputted, the material on the plate generates a pattern.

Jenny then proceeded to invent the tonoscope, which was constructed to make the human voice visible. By singing into a pipe, the air passes through, causing vibrations on the black diaphragm, which has quartz sand evenly spread across it.

Hans Jenny stated that if you had the same frequency and the same tension, you would get the same form, with low tones generating simple patterns and high tones resulting in more complex designs.

The pattern is characteristic not only of the sound but also the pitch of the speech. Hans Jenny also used this device to visualise music, namely orchestral music such as Bach and Mozart.

    Thomas Wilfred was born in Denmark in 1889. Upon moving to New York in 1919, he co-founded the Promethans, who were a group dedicated to exploring spiritual matters through artistic expression.

Speaking of light as an art form, 1922 saw Wilfred invent the Clavilux, which was considered to be the first device designed for audio-visual shows.

    The Clavilux had six projectors, which were controlled by a keyboard consisting of banks of sliders, which would resemble a modern lighting desk. An arrangement of prisms would be placed in front of each light source. Wilfred mixed the intensity of colour along with a selection of geometric patterns.

    Although most of Wilfred`s performances with the Clavilux were presented in complete silence, it was not until 1926, that he collaborated with the Philadelphia Orchestra in the presentation of Rimsky-Korsakov’s Scheherazade.

    Thomas Wilfred produced roughly forty works before his death in 1968 but only eighteen pieces have since survived. The Clavilux was capable of creating complex light forms, which mix together to create a depth of light; this could be seen as resembling the northern lights in Iceland.

Influenced by Thomas Wilfred`s colour organ and Leon Theremins music, Mary Ellen Bute began to develop a kinetic visual art form. She produced several abstract animations set to classical music by Bach and Shostakovich.

    This was achieved by submerging tiny mirrors in tubs of oil and connecting them to an oscillator. With the production of these animations, Mary Ellen Bute said that she sought to “Bring to the eyes a combination of visual forms unfolding along with the thematic development and rhythmic cadences of music”.

She referred to some of her films as seeing sound and a few of Bute`s abstract films were shown at Radio City music hall and were often screened before Hollywood feature films. Center for Visual Music. (2014)

    In 1921 German, painter and filmmaker Walter Ruttmann created Opus 1. He assembled each projection print of the film with an old college friend who wrote the score. A string quintet performed live with each screening if Opus 1, which was shown in several cities across Germany. The abstract shapes moved onto the screen in time with the music, Ruttmann achieved this by drawing colour pictures in the musical score so musicians would be able to synchronise their playing with the film.

    Upon the attendance of a rehearsal of Opus 1 in Frankfurt, Oskar Fischinger decided to make visual music.         He started to experiment with slicing wax and clay images while using silhouettes combined with drawn animations.

Fischinger made some of his earlier films using a colour organ which was controlled by several slide projectors and stage spotlights that had changing colour filters and fading capabilities.

    In 1925 he designed a new colour organ with five projectors, which added a more complex layer of colour. Fischinger created wooden cubes and cylinders that were painted and coloured with fabric, that were projected on screen to create his films.

When moving to America, Fischinger created great works such as “An Optical Poem” which was set to the music of “Hungarian Rhapsody no.2” and “Motion Painting no. 1” set to the music of J.S Bach`s “Brandenburg Concerto no.3”.

When attending the “Art in Cinema Festival” in San Francisco in 1947, Fischinger met two painters who had been inspired by his work. Harry Smith painted directly on the filmstrip and the resulting film was accompanied by a jazz performance.

Jordan Belson, in 1957 began to choreograph visual accompaniments to new electronic music. Composer Henry Jacobs composed the electronic music while Belson created the visuals using multiple projection devices.

    In 1961 he began to create live visuals by the manipulation of pure light. Taking the role of a modern VJ, with his use of custom built optical bench with rotary tables, variable speed motors and lights of varied intensity, he would create live visuals to accompany electronic music.

Belson did not want any of his material uploaded online; therefore not many of his works are available. Norman Mc Laren was born in Scotland in 1914, while studying art and interior design at the Glasgow School of Art in 1933; he began to make short experimental films,

    Mc Laren wrote that while listening to music he would see abstract images in his mind and after watching his first abstract film in 1934, he discovered a way in which he could make these images in his head visible to others through film.

By painting onto film cells, he had the ability to display a visual representation of music.

    Incorporating a variety of musical styles into his films including Indian music by Ravi Shankar, Trinidadian string band and a jazz piano soundtrack by Oscar Peterson.

Mc Laren also used a technique he called “Animated Sound” by scratching directly onto the soundtrack of the film, he would create unusual electronic sounds and this can be heard in his film entitled “Blinkity Blank” from 1955.

While an undergraduate student in electronic engineering and electronic music at the university of Illinois, American video artist Stephen Beck first began to experiment with the use of video and electronic wave forms to create images. In 1969, the Beck direct video synthesizer was designed; this device would construct an image using the basic visual elements of form, shape, colour, texture and motion. Using no camera Beck`s invention would generate videos from sound.

In his essay titles “Image Processing and Video Synthesis”, the video artist discussed that the four distinct categories of electronic video instruments are:

Camera Image Processing

Direct Video Synthesis

Scan Modulation/Rescan

Non-VTR Recordable

The Camera Image Processing was used to modify signal to a black and white television camera by adding colour to its signal.

Direct Video Synthesisers were designed to operate without a camera, containing circuitry to generate a complete video signal which included colour generators to produce colour, a form generator circuitry which was designed to create shapes and motion modulation to move the shapes through electronic wave forms such as curve, sine and other frequency wave patterns.

Scan Modulation/Rescan was used to manipulate images by means of deflection and electronic modulation, images on the screen can be rotated, stretched and reflected.

Non-VTR Recordable is a TV to display his output. Stephen Beck. (1975).

In 1973, a series of live performances took place titled ”Illuminated Music”. With Stephen Beck controlling the visuals and electronic musician Warner Jepson using the Buchla 100 analogue modular synthesiser while performing the music to accompany the visuals.

Both Beck and Jepson who were members of the National Center for Experiments in Television worked together, performing Illuminated Music in front of audiences in Dallas, Boston and Washington DC.

These performances demonstrated the integration between electronic music and video synthesis, an art form, which is still used to this day.

The majority of electronic music concerts have a visual element present, this is either performed by the artist themselves or more frequently by video programmers who will tour with and work with the artist in question in developing and performing the visual element of the performance.

Commonly used software for this is Resolume, VDMX or Mad Mapper.

With Graphics Processing Units (GPU) and processors getting more powerful over the years, many modern methods to develop and programme videos were made available. Quartz Composer, Jitter and VVVV are all video synthesis tools used to create original videos.

Famous Colour Music Artist 

Oscar Fishinger

Oskar Fischinger was born June 22, 1900 in the small town of Gelnhausen, Germany, his family owned a drugstore and a brewery-restaurant. 

WWI Fishinger was called to combat duty. Due to his health issues he was rejected from the army. After WWI Fishinger attended a trade school and got an Engineer’s Diploma.

    In Frankfurt 1920, Fischinger met Bernhard Diebold a critic at a literary club , who later in April, 1921 introduced Fischinger to Walter Ruttmann, a pioneer in abstract film. At the time Fischinger was experimenting with colored liquids and 3D modelling materials such as wax and clay. He invented a "Wax Slicing Machine", which synchronized a vertical slicer with a movie camera's shutter. Fischinger kept in contact with Ruttmann. In August, 1922, Fischinger resigned his engineer's job and moved to Munich to become a full-time filmmaker. He later on  licensed the wax slicing machine to Ruttmann, During that time Fischinger shot many abstract tests of his own using the machine. Some of these abstract tests are distributed today under the title Wax Experiments.

    In 1924, Fischinger formed a company with American entrepreneur Louis Seel to produce satirical cartoons that tended toward mature audiences. One survives in his film estate, Pierrette I. He also continued to make abstract films and tests of his own, trying new and different techniques, including multiple projector performances. In 1926 and 1927, Fischinger performed his own multiple projector film shows with various musical accompaniments. These shows were titled Fieber (Fever), Vakuum, and Macht (Power). He also collaborated with composer Alexander Lazlo on Farblicht-musik, a multiple-projection light show which toured theatres and exhibitions throughout Germany. By June, 1927, financial difficulties related to the failure of the Seel Co. and general inflationary troubles forced Fischinger to leave Munich, so he walked to Berlin where he re-established himself with difficulty.

     In July, 1928, he went to work for UFA doing rockets and other special effects for Fritz Lang's science fiction feature Frau im Mond.

A year later he accidentally broke his ankle at the UFA studios and while hospitalised, decided that this must be a sign for him to devote himself full-time to abstract filmmaking which he then did, producing over the next three years the remarkable series of black-and-white studies tightly synchronized to music. These Studies, screened widely in Europe, Japan and America, came to be in such demand that by 1932 Fischinger had engaged his brother Hans, his wife Elfriede, and three other girls to work at Fischinger Studio.

This also freed him to pursue experiments with drawn synthetic sound and to collaborate with Bela Gaspar on the development of a three-color subtractive tripack film and camera process, GasparColor, which allowed him, in December, 1933, to complete his first color film Kreise. 

Fischinger's subsequent color films Muratti Marches On and Composition in Blue gained so much critical and popular acclaim that Paramount offered him a contract, and in February, 1936, he set sail for Hollywood

never to return to Germany.

    Fischinger's independent temperament and language difficulties made it extremely hard for him to work in studio situations, although he endured episodes at Paramount (1936), MGM (1937), and Disney (1938-9), and later (1941-2) sat out a stint with Orson Welles for an unrealised project. His frustration at not being able to produce independent film as he had in Berlin led him to take up oil painting, and through the success of his canvases he came under the patronage of Hilla Rebay, curator of the Solomon Guggenheim Foundation, who extended several grants to him during the difficult war years. Unfortunately, Fischinger and Rebay quarreled over the artistic merits of his film Motion Painting no. I (1947) and he never again received adequate financial support - from Rebay or other sources - to complete another film. For the last twenty years of his life, Fischinger had to content himself with endless unfulfilled projects, with his paintings, with a home light-show instrument, the Lumigraph, which he invented in the early 50s

but for which he never found commercial distribution.

After some years of relative ill-health, he died on January 31, 1967. 

bottom of page