1.Description

Transparency/Opacity is a work of interactive art using the mediums of sound, light and video to create an immersive space that evokes a dreamlike state.  It is a synthesis of two themes; the first being the idea of the hidden mediation of the human computer interface as discussed in the paper Transforming Mirrors by David Rokeby; the second being the surreal feeling evoked by the memories of dreams as personally exemplified by the statement “the memory of an echo from a dream,” a thought I’ve meditated on for many years.

The piece itself plays on the margin of transparency and opacity; transparency of physical materials, transparency of the software and audio/video interface, and intersubjective transparency.  The transparency of the the physical materials is embodied in a combination of a water table and an overhead screen.  The water pan is a fixture four feet high, four feet long and one foot wide, with a shallow pan containing water with a transparent bottom.  The screen is a rectangle of proportional dimensions (6′ x 18″) hung horizontally over the top of the water pan.  A small LED light placed on the floor below the transparent pan shines up through the water and lights the overhead screen.  As the water is disturbed graceful lines of darker and lighter lines are formed on the overhead screen, tracing out patterns of diffusion and interference on the relatively opaque surface.

The transparency of the software and audio/video begins on the opposite side of the screen.  Placed above the horizontal screen is a camera that records the patterns of light projected on to the screen from the water table.  This video is taken into the software, where it is broken down in to matrices of information.  From these matrices various means of control are extrapolated.  For the video screen, the video of the water is combined with a video of a monologue.  The water ‘overlays’ the monologue, partially obscuring the face.  As the water is disturbed and the patterns of light and dark are generated, the brightest points of light are processed to be transparent, thus as the water is disturbed more of the face is revealed; as the water moves toward opacity, the screen moves toward clarity.   Similarly for the audio, certain points of the matrices are monitored for average brightness; when that brightness exceeds a certain threshold, the muffling filter over the voice is released, revealing the words of the monologue and triggering the sound of falling drops of water.  As the level falls below the threshold, random synthesizer tones and more water noises are triggered, the time and direction of the monologue are changed and the muffling filter is returned.  As the water moves toward opacity, the audio moves toward clarity.

The intersubjective transparency is in the monologue and the interaction.  The monologue is a recording of myself speaking quite candidly about deeply personal subjects, things I never discuss with anyone, sometimes not even with myself.  As the experiencer engages with the installation these secrets are furtively revealed, but only in short bursts, usually quite unrevealing in themselves, but the tone and the facial expression also communicate meaning.  This stands as a metaphor for all interpersonal interaction, as we stir each others surface responses, deeper things are revealed if one is paying close attention.

 

2.Influences

2.1 Transforming mirrors

In Transforming Mirrors David Rokeby discusses the perceptions of transparency in human computer interaction, how it is different for the interactive artist and those creating interactions for commercial purposes, and how tacitly accepting the created interactions can have less than innocent implications for the user and society as a whole.  In commercial human computer interaction (HCI), transparency of the interface is the ultimate goal.  The creator of the interface wants it to seem to disappear, putting the user in direct contact with whatever content that they seek.  But this is a false transparency; the interaction, though it may seem natural, has always been mediated by the creator of the interface; implicit in its creation are the creators goals, biases and blind spots.  “By filtering out apparent irrelevancies, giving us us simplified representations of our relationships, interactive media make it easier for us to make decisions. These filters operate like a belief system.”  But the results of this tendency are dire: ”When we forfeit the right or power to decide for ourselves the nature of these systems of generalization, we commit ourselves to an ‘objectivized’ point of view which is entirely in the control of others…” 

Transparency/Opacity draws from this state of affairs by repeatedly reversing clarity with obfuscation by the means of various filters; be they the physical type in the direct water/touch/light interaction, the technologically mediated type in the software,  or the personal type where the intimate details are carefully curated via technology.  To gain clarity on the screen, for the monologue to be clear, the water must be unclear.  To enjoy the interesting patterns on the overhead screen, one must stop agitating the water, and thus lose any chance of understanding the monologue.  Where the window in a normal HCI interaction is an impermeable piece of glass, ie a screen, in T/O you must break the membrane of the surface of the water in order to start the interaction; when you are done interacting, a bit of the water, the interaction, remains on your hand; it is not a clean separation.

2.2 The memories of echoes from a dream

As I stated previously in this blog post (The Humidity of Emotion -or- That Feel), my intention with this piece was to avoid the simple mechanism of ‘technical accomplishment as a vehicle of audience engagement’ to a more organic, emotionally resonant work that involved a high amount of technology but is done in such a way that the technology disappears and one is left in an ’emotionally humid’ environment that does not dictate the feeling of a particular emotion, but encourages an emotional response.  (This disappearance of the technology follows, in a way, from Rokeby.)  The dark environment, the obscured speech and face, the interaction with an archetypal element (the water), the beautiful but momentary patterns of light, were all meant to invoke interactions in dreams and the strange moment when a person awakes and immediately finds themselves reflecting on the dream.  Here again, transparency and opacity come into play; what is it that you remember?  What stuff is that made out of?  What is the memory of a dream? What is the memory of an echo in a dream?  Are you actually remembering the dream how it was, or is there now another layer of interpretation from you conscious mind altering the memory? (Is there a filter or is it transparent?)

3.Technical Considerations

3.1 Video Filtering

The effect of the water obscuring the monologue was achieved by using the technique of luma keying.  Luma keying is the process of reading the luminescence value of a particular pixel in a video matrix and acting upon that pixel if it is above or below a certain value.  The luminescence of a pixel is a calculation of the input RGB values of a pixel against a scalar; lum = (ascale * alpha_value) + (rscale * r_value) + (gscale * g_value) + (bscale * b_value) from the Cycling 74 wiki. When the value  of a pixel from the camera feed exceeds the luminescence key, the alpha (or transparency value) is set to zero, thus allowing the background video, which is the monologue, to come through.  This is accomplished using Max MSP Jitter and the jit.lumakey object.  The object is used twice in the patch, once for the video filtering, and once purely as a data source that is then used by the audio patch to adjust the filtering.

3.2 Audio Filtering and Memory adjustments

As the project increased in complexity, it began to tax the processor of the computer.  The first step was to move the video from the jitter matrix to the jit.gl.videoplane, which moved calculation of the video from the main processor over to the video card.  When this was still not enough, it became apparent that the audio had to be handled more efficiently.  This was accomplished by encapsulating several of the sample file players (sfplay~) into poly~ objects.  The poly~ object moves audio generation out into multiple threads, as well as killing processing while a particular voice is not being used.  Here are the images of the poly~ object and it’s internal structure.

 

 

Max file is here.