In the beginning of twentieth century, the idea of correlating sound and visual forms became more compelling; from clavier à lumières of Alexander Scriabin to Arnold Schoenberg-Vasily Kandinsky encounter. However, it was only with the advent of new technology in the second half of twentieth century that the possibility of real-time interaction and synthesis of sound and visual has become at hand. Today, the so called audio-visual art is the result of coming together of two old classical worlds, visual and sonic, through mediation of digital language. Digital language itself has made possible a translation of all sorts of signals into each other i.e. we are now able to visualise or sonify all sorts of raw data. This phenomena is new. But, before asking what this new possibility of mathematically mapping all sort of sign waves into each other, translating electromagnetic waves into acoustic waves and vice versa, can bring us, we should examine what cognitive capacity, what perceptual potential, is engaged. Is it only people with synesthesia who experience a blend perception? Or all our cognition is in essence embodied and thus linked? When all symbolic meanings is put aside, what is the immediate experience of audio-visual?
 
Audio-visual art offers an embodied unified sensory experience that alters our conscious and redefines space. In respond to blatant question of whether audio-visual is subject for our hearing or our seeing, one should strongly answer both at the same time. In audio-visual one cannot separate sound and visual from each other without disrupting the essence of this new art, this is the measure of what audio-visual is. In the audio-visual art, the relation of sonic and visual stimulus is no longer a simple one, but a new language is created for decoding, decrypting, deciphering, revealing and then again concealing, reinterpreting, imagining, evoking a fleeting moment of what might present itself as the real or as the true. This new language articulated by intertwining space, sound and image has now become medium as well as mediator of new ways of dialogues.

This project is an experiment of this new dialogue. This piece blends two mediums together, sound and visual, so that these two play with each other. The visual part is result of an artistic pursuit and painterly experience that has transformed painter's mind set. This project is portraying this new mind set. That is why we name it “Refactor”. The core of this project is a computer program based on evaluating, reviewing, reflecting and reinterpreting painter's practice of art making. The outcome of this self-reflection is turned into compositional algorithms that can be formulated mathematically, so that their sequential outcome forms a dynamic network of computer codes. These computer codes are making possible painterly experience without the painter, a painting factory without any need for painter's presence or control. These computer codes recreate process of image making and control painterly quality of this realization on dedicated visual platforms. 



Nikzad: After series of painting exhibition, for Nikzad Arabshahi, the basic idea behind visuals of this project is to reconstruct his painting in real-time through computer rendering in a generative algorithmic composition. All of these algorithms are implemented by Nikzad Arabshahi in Derivative Touchdesigner. Refactor is culmination of his earlier painting projects such as (Eraser collection – 2013) and (Chlordiazepoxide collection – 2011), that is now transformed into this generative art project. For this mean, various sensors like Kinect, Leapmotion, EEG, Myo and Wacom have been installed in his studio capturing his hand’s physical movements and different ways he uses both physical painting brushes as well as virtual brushes in digital painting for two consecutive weeks. Analysis of these bundles of data led to design of Touchdesign modules imitating his behaviours. After this, charts of materials science & technical application of Acrylic paint has been employed for modelling and simulating painting techniques like Impasto, Glazing, wet on wet, brushwork, etc. The outcome of this simulation is visual elements and units in the form of lines, shapes and marks, but for assembling these outcomes into meaningful shapes and patterns another layer of local compositional algorithm for data is devised, at this point a network of modules is working in such way to build a meaningful texture. After this, at the final stage, another compositional algorithm for general composition of the whole picture is added. This network is condensed and becomes a drawing engine. Five of these simulation engines are used for reaching the final picture. 

Operation sequencing of each painting simulation engine can be summerized as follows:
1- Analysing and resampling main data, design a channel operators network system to simulate basic algorithms using various CHOPs to generate and control data.
2- Using CHOP’s data to design and control various Surface operations (SOP) and Particle systems to render basic visual elements.
3- Using SOP’s renders to redesign images by making Texture operation (TOP) networks for final composites.
4- Using TOP to CHOP techniques for image processing 
5- Mapping and translating final image into parameters meaningful for data sonification in two separate channels. These parameters are: Red, Green, Blue, Alpha, and Contrast of the final image, Playfulness and intensity of movement and performance of SOPs and particle systems in x-y-z units. These data are filtered and smoothed by 7fps and sent to Max/MSP through the UDP protocol for generating sound.

The final image is mapped into a time-based format and is projected onto two raw canvases of size onto two raw canvases size 140 x 236 inch hanged on the gallery’s wall and three metal multiple layers sculpture size 141 x 149 x 188 inch installed three sides of gallery’s space. The visual platform Refactor is an interwoven spatial-visual-aural sculpture that redefines space through arrangement of generated recurring images and sounds; images that go beyond conventional static white walls of a gallery and travel through these platforms. Images passing through layers of semi-transparent semi-reflecting metallic net are multiplied in space. Visual images are immersed in the soundscape and sound are emerged from visual images. Painting become sound and fade away from our sight like sound.

Vedad: Meaningful data is not overabundant. If one takes raw 1920 x 1080 px in 30fps visual data, that is a bitrate about 10Mb per second, this abundance of raw data does not give meaningful parameters for understanding a picture. The whole complex mathematical methods of pattern recognition are in fact practice of designing algorithms to translate these abundance of raw data into few meaningful parameters. After Second World War, composer has been using graphical score for their new conception of music, Articulation of George Ligeti is a famous example, but some graphical scores like Earle Brown’s Folio does not give a one-to-one relation between the score and the final sound and leaves more room for free interpretation. 

Here in Refactor, we have adopted a time-based right to left movement as the base of for the score, the vertical axe is divided into a grid of three octaves of microtonal just intonation intervals that is outcome of extensive research into forgotten musical intervals of Qutb al-Din al-Shirazi (1236 – 1311). A set of three sin wave / white noise modular synthesiser is implemented, within the main one an oblique line, say going from low to high in certain time, will sound a glissando on this musical scale, the second one at octave sliding freely between notes producing a chirp sound, and the third one moving in triple time. This set is then layered into itself with help of poly~ object in Max/Msp to build a polyphonic texture that creates a soundscape that bring the visual to life. The visual’s colour pallet and its contrasts determine intensities of harmonics of these sets of modular synthesisers. And an extra parameter named playfulness is defined to map how much activity is within the visual, that is how much the visual is chaotic, playful and noisy or how much it is coherent and ordered. The final soundscape made of complex series of sin wave and white noise modular try to give general feel, a general atmosphere of the visual.  Sound moves in space based on how much each projector is active. At the end, audience is immersed in a sonic environment, in a soundscape that would put him inside the generated visuals, as if living in it, as if meditating in a temple of sounds and colours and movements. 

Written by: Vedad FamourZadeh, Nikzad Arabshahi

Add

Nikzad Arabshahi