malfunction

"Effective operation is done without awareness. Or we can define awareness as that which happens when we malfunction"

- Stelarc

Awareness as a malfunction


Often times people struggle to find the right words to describe where or exactly how something feels in their bodies. Most lack sufficient knowledge of anatomy to point out to their partners that it itches most between the rhomboideus and the latissimus dorsi. Instead humans find bits of their bodies through manual reconnaissance. They scratch and scratch until the precise point where it itches is found, only then do they find relief.

Lacking the verbal capacity to describe bodily sensation is a limitation but not a vital impediment. The body operates quite effectively while sensation happens below the threshold of awareness. It is only when the body stops functioning normally or when sensorial input contradicts sensation that awareness sharpens and what before was unconscious now surfaces as a concern. It is then that words seem lacking.

It is this feature of body operation that I seek to exploit. In this light, a malfunction could be introduced artificially, that would help the subject bring proprioceptive sensation to the level of conscious awareness.

Hackable Senses


The human sensory system is composed by the terminal points of millions of nerve fibres that interpret and carry information to other parts of our body. This information is the result of concrete stimuli, as each of the nerve cells that compose the sensor reacts only to a particular kind of stimulus. A photoreceptor in the eye for example can only be stimulated by one type of light. Human beings have four types of photoreceptors in their retinas. The aural receptors in our inner ear cannot be stimulated by light, they only transmit information when stimulated by sound. Whatever the actual interpretation the senses might give to stimuli, what is clear though is that "each nerve can only respond to stimuli in a specific way; therefore our knowledge of the world reflects the structure of our nervous system".

An approach to tamper human sensation is by providing fabricated stimuli to the nervous system in the specific ways in which it can process them. This way it is possible to fool the sensory system into processing sensations that do not come from real-world phenomena but from a machine simulation. This is exactly how the controlled malfunction that is proposed in this paper works.

The human nervous system receives as input the signals that a computational process produces as output.

My interest in this research is not rooted only in experimenting with human-machine interfaces, but to better understand the cognitive processes of perception. A lot of knowledge exists on visual and aural perception, but during my research I found significantly less information about kinesthetic perception and its role in consciousness. Curiosity naturally lead me to dig deeper into it.

Hacking Proprioception


"The inner sense by which the body is aware of itself"

"An Anthropologist on Mars", 1995, Oliver Sacks



The ground below the feet is the most important reference point to the body's sense of space, the contact of the feet propel the body upwards and keep it in dynamic balance. Humans sometimes mistake perceptions of their position relative to the Earth. You might have experienced this when you are sitting in a stationary train and the train next to you starts moving. For a few moments it is hard to tell whether you are moving or standing still. This kind of spatial disorientation occurs when humans receive contradictory information from the variety of senses that contribute to the perception of position in space.

Some humans get nauseous when they read in a bus, their eyes are fixed on the text of the page that moves along with them and so it appears stationary to the eye, yet the vestibular system is perceiving the motion of the bus and of the body with respect to the Earth. Many people get dizzy as a consequence, although sensitivity to these sensations vary among different people.

A dancer explained to me that when she goes into an old wooden house, where the floor and stairs are uneven and at varying angles, she gets a very unpleasant sensation in her stomach followed by a strong desire to leave the place. She can feel the disorientation caused by conflicting sensorial sensations and her expectations.

The body's perception of itself is mediated by the proprioceptive sense. This sense uses input from different parts of the body to determine its own position and posture. Skeletal joints, ligaments, muscles as well as the vestibular system in the inner ear, hearing itself and vision intervene in the body's understanding of its own position in space. The system formed by these inputs is redundant, this means that two of these systems can be providing information about the same sensation. When one of these systems provides information that conflicts with information provided by another there's a momentary cognitive dissonance. This dissonance manifests itself by raising awareness, what was happening below the line of awareness before, becomes a conscious cognitive process.

In an anechoic chamber, for example, is impossible for the body to determine how close or far it is from a wall, these kinds of chambers have specially designed walls and floors where sound becomes trapped and cannot bounce back. So the human ear cannot appreciate the room's acoustics.

In microgravity it is impossible for the body to tell which way is up. An external reference point that can be identified visually is needed.

By providing artificial input to the vestibular system it is possible to induce in a human a sensation of spatial disorientation. There are several approaches by which this could be achieved, but I will focus on the one that is specific to my work.

The Vestibular System


This complex system consists of three semicircular conduits located in the inner ear, each providing sensations for its orientation and motion with respect to one axis in three-dimensional space. So for example the movement one would use to say "no", rotating the head sideways, would provide stronger stimulation through one of the conduits, the one that sits in the horizontal with respect to the floor; the stimulus would be less strong in the other two conduits. The actual science is rather more complex than this explanation would indicate but my point is that this organ detects forces by sensing differentials in the sensations provided by these conduits.

The three conduits converge on two receptacles inside the inner ear called the saccule and the utricle. Inside of each of these, there is an organelle called the macula, which acts as a sensor and can measure forces through nerve terminations that are attached to hair-like structures that are suspended in a semi-liquid membrane.

These smalls differences in sensing one axis of movement over the other, give the vestibular system a measurement of the forces that the body is subject to, which in combination with other senses give us a pretty accurate understanding of how our body is moving in space.

The vestibular system is also principally responsible for balance. Abnormalities in vestibular sensation can have a severe impact on a person's balance and capability to orient itself during bodily motion to the point of serious impairment. People suffering from Ménière's Disease know this all too well. First described by physician Prosper Ménière, vertigo can be caused by inner ear disorders such as infections.

These sensations can also be induced artificially through several techniques such as injecting hot water into the ear followed by cold water, Epley’s maneuver which is a therapeutic exercise or a special type of electrical stimulation named GVS which is the one that concerns this research.

What is GVS?


The technique more broadly known as GVS was discovered in the eighteenth century when Alessandro Volta reported feeling dizziness after he applied current through his head. Volta, inventor of the battery, put electrodes attached to a 30 Volt source and applied them to his ears, resulting in what was described as "an explosion inside of his head, including loud noises and disorientation".

Since 1820, when Johan Purkyne described the technique in his dissertation, GVS has been used as a technique to study the function of the vestibular system.

GVS stands for Galvanic Vestibular Stimulation and it consists of the transcutaneous stimulation of the vestibular apparatus. The vestibular system is situated in the inner ears, both left and right, and is responsible for providing the brain with information about orientation and inertia. The brain then uses this information in combination with visual and aural input to determine the position of the body in space.

Artificial stimulation is achieved by placing electrodes on the skin covering the mastoid processes (fig 5) of the temporal bone. You can feel these protrusions if you touch a human skull just behind the ears. Creating an electrical field between these areas on both sides of the skull produces a very coarse stimulation of the vestibular system. The mastoids are just bellow the brain, so current doesn’t go directly across the brain.

The nerve terminations in the maculas are sensitive to changes in this electrical field and the vestibular system tries to compensate with a response in the sensation of balance that makes the subject lean towards the anode (usually marked +).

Early experiments


The circuitry required to induce GVS is in fact very simple, all one needs is a way to regulate the current (amperage) so that it stays within non-painful thresholds (below 5mA) and place the regulating circuit between the source and the electrodes.

Once I had a primitive circuit that could produce vestibular stimuli I connected it to a microcontroller and wrote a piece of software that could play back a fixed sequence of stimuli. I experimented on myself for quite a while, performing different activities such as walking along a line, drawing two parallel lines on the floor, standing on one foot with eyes closed, tip-toeing along a path, executing repetitions of balancing exercises such as drunk tests, tai-chi forms, running, spinning, balancing on a fence, standing upside-down on my hands, standing on a gym exercise ball and many more. I tried many of these activities to understand the effects of the stimulation on my own perception of balance. I recorded some of these exercises to try and see if these effects could be visible by an external observer.

My findings after these experiments proved that the experience was rather consistent. GVS is strongest with eyes closed but most confusing with eyes open. Repetitive tasks make the effect more obvious to an external observer. The effects are very subtle, it is hard to tell when someone is under the influence of GVS unless there is some comparative framework. All activities become much more challenging when under the influence of stimulation. For example, drawing two parallel lines on the floor about 10 meters long, took about 1’ 30” without stimulation and almost 3’ with stimulation. Both trials resulted in properly drawn parallels, but analysis of the video showed more postural jiggling, more resting pauses and more general postural discomfort on the trial with stimulation than on the one without.

In the following two illustrations Fig 6 and Fig 7, a point on the head was tracked in a video capture of the author executing a simple activity, drawing two parallel lines on the floor. The blue track shows the path of the head without GVS, the red path shows the path of the head under the influence of GVS.

The effects of GVS


A subject walking in a straight line will deviate their path when subjected to GVS and will lean slightly towards the anode in an effort to compensate. I would describe the sensation as being in a boat and feeling that the floor beneath the feet has tilted slightly. The deviation is subtle but externally visible to an outside observer, the subject appears to be out of control of their motor capability, very much as if the subject was drunk and incapable of walking in a straight line.

A human can function normally under the effects of GVS but to do so must use considerably more effort as every move becomes challenging to execute with precision.

When stimuli follow each other too quickly or alternate directions, the sensation will induce a mild nausea in most subjects.

The sensation caused by GVS isn’t pleasant as it is somewhat disorienting. When levels of current are limited to reasonable thresholds (3.5mA), the sensation is not painful but still unpleasant.

One of the pioneering researchers in GVS as an interactive interface, Taro Maeda, has published several papers that infuse with great enthusiasm fantastical applications to this form of stimulation. My personal experience with it has led me to think that one shouldn’t expect popular uptake of this kind of technology any time soon, as it is unreliable, requires careful calibration and when functioning correctly it still results in a feeling that is quite unpleasant.

Getting used to it


I have been subjecting myself to GVS for a little over a year and a half now and I have developed an interesting somatic response to the sensation that repeats consistently. As I subject myself to GVS I feel the tensional patterns in my body compensating for the sensation, this results in a subtle stiffening of my posture starting on the neck in an attempt to feel steady. My body has learned a new habit from electric stimulation.

Over the course of my research my sense of balance has developed greatly and I can now detect subtle changes in the tensional patterns of my body. I cannot give full credit to the GVS for this improvement, as my research involved learning other techniques that have these changes as their stated goal such as the Alexander Technique. But GVS has been instrumental in testing my progress and has been decisive in bringing the sensation of balance into conscious thought so that I could analyze my own movement through proprioceptive feeling.

My research led me to a deeper understanding of proprioceptive sensation and a method to manipulate it with some consistency, but until this point I had been preoccupied mostly with reproducing existing knowledge. As a visual artist it was frustrating to me that proprioceptive sensation was so difficult to communicate and share with others. Indeed as dance theorist Barbara Montero once wrote on its aesthetic potential:

"Proprioception poses a unique problem: aesthetic senses seem to require a distinction between the object one senses and the bodily sensation itself, a distinction that can be made with sight, smell, taste, touch and hearing. However, proprioception, it might be claimed, cannot focus our attention beyond ourselves"

"Proprioception as an Aesthetic Sense", Barbara Montero



To increase awareness of proprioceptive sensations, I try to induce a malfunction. This malfunction is regulated by a computer controller and the effect I seek to have is to inhibit and direct movement. I use GVS as a technique to produce dissonance between what is felt and what is seen.

But could this feeling be shared in any way? Could someone “proprioceive someone else’s movement” as Barbara Montero puts it?

The BRAID system


BRAID is an autonomous wearable system composed of two devices designed to extend the sense of proprioception of two actors. Each device registers the wearer’s position and broadcasts it to all the nodes in the mesh network on the stage. As a response, the other device produces small electrical pulses that stimulate the vestibular system of the wearer, altering their perception of balance. When one of the actors leans very heavily towards the left, for example, the other will feel a proportional stimulation towards the opposite side.

This system was designed by speculating what a shared feeling of proprioception could feel like and building a new medium for proprioceptive sensation from existing techniques.

The system is composed of custom-developed electronics for transcutaneous stimulation and the computing platform is based on off-the-shelf parts like Arduino and XBee wireless networking modules.

The devices


The BRAID devices are encased in a box with a posterior lid shaped as a belt buckle that can be easily mounted on a strap. Each of these devices can keep track of head movements through an accelerometer mounted in a headband and connected to the device through a flexible five lead ribbon cable. The two additional leads are outputs from the stimulation circuit that are connected to two medical electrodes that the wearer applies on the skin that covers the mastoid processes. The connectors (in red in the picture) are hand made as they went through many redesigns during the prototyping phase.

The intensity of stimulation can be manually regulated through a potentiometer on top of the device. This is an essential part of the design as the device requires frequent recalibration by the wearer, as skin resistance varies through the duration of a session.

These devices run on a 9V battery that serves as power source for all the components inside as well as the transcutaneous stimulation. They run for about 2 hours on a single charge.

Postural Zero


In the first five seconds after the device is turned on it tries to figure out the center balancing point by averaging thousands of measurements of the wearer’s position in a calibration loop. So if the person is in a normal standing position, the device will understand that position as the Postural Zero. Or if the person is laying on their back, that will be the Postural Zero and therefore standing up will result in a deviation from the Postural Zero.

Large deviations from this Zero Position will result in stimulation in the other devices in the network. This feature of the system allows actors to change their Postural Zero during a performance, or try different starting positions during rehearsals.

Inputs and outputs


The system gathers data about the dancer’s position with respect to their Postural Zero and uses those deviations in two different ways depending on its mode of operation which can be configured through a switch in the front of the device.

Each device has two kinds of outputs, status and stimulation values. The status values comprise information such as battery charge, mode of operation, current and voltage readings and the stimulation value is a single signed value that indicates polarity and strength of stimulation.

All inputs and outputs can be processed by a monitoring node in the network to produce visualizations or sonifications of the data. This is useful for live performance as each of the parameters transmitted can be mapped to a visual or sonic feature in the scenography.

Modes of operation


Each device has two modes of operation, manual in which the input gathered by a single device is not broadcast and is used by the device itself to determine the stimulation that its own wearer will receive. So the wearer can stimulate itself, this mode is often used for testing or exercising.

The other mode of operation is auto. In this mode input data from the accelerometer is broadcast to other nodes and each node processes all data except the self generated. This is the mode that permits that the inclination of one actor will result in the stimulation of another.

Networking


Each device is fitted with an XBee module configured in mesh mode. This means that each device can communicate with every other device within range. The current system supports two devices only due to budgetary constraints, but each device contains a unique identifier and the system can scale to many more devices.

The status of the network is refreshed at a frequency of 28Hz, a slightly slower rate than the rate of operation of the device and this does not include latencies derived from poor signal or low battery levels. The range is about 20 metres with line of sight to about 6 meters without. They work quite well on a stage environment. When the system is operating in a mesh node it doesn’t have the capability to connect to IP networks, but adding a monitoring node (e.g. a computer with an XBee explorer module) could relay all traffic in the mesh network to an IP network.

My personal computer serves as the monitoring node in the current system, but it is not needed for normal operation on the stage.

Levels of stimulation


The device can provide levels of stimulation that can vary from 0 to 3.5mA with 0 to 24V of potential. The maximum cut-off current is 3.5mA which was advised as a reasonable level to remain medically safe, the stimulation circuit never exceed this limit. This limit can be pushed down if it’s too strong for the subject. The device contains an internal potentiometer that can bring down the current limiting factor.

when the system is engaged in activity the dynamic level of stimulation is computed as a factor of the inclination received from another device. Through the following factor

Pstim = Pmax (Iinput / Iref)3

Where Pstim is the value of stimulation, Pmax is the maximum stimulation that the system is capable of, Iinput is the value of inclination received from the other node and Iref is the normalized mean inclination. The cubic formula insures that very small variations in inclination will zero out, while significant variations in inclination will result in more pronounced stimulation. This formula was determined experimentally through trial and error, until the level of stimulation “felt right”.


fig. 9 - correlation between inclination (horizontal) and level of stimulation (vertical)

This formula means that the receiving actor doesn’t feel anything much at all until the one inflicting hasn’t moved about 50% away from their Postural Zero.

The level of stimulation is determined by a microcontroller by using a pulse width modulation (PWM) signal running at 31.25kHz. The microcontroller uses easing algorithms to smooth the signal in and out. This is necessary because sudden changes in electric stimulation through the skin often have nefarious effects on it, causing decomposition, irritation and even pain if the changes are too sudden. Smoothing the stimulation signal can avoid some of these problems.

Record and Playback


The BRAID system has the capability to record sessions, logging all positional data of all the actors and is capable of playing it back in the form of vestibular stimulation on another subject. This opens the possibility, for example, to share proprioceptive sensation through a recording of an event in the past. Or across geographical areas through IP networks where the actors can be much further apart than they could on the stage.

Dealing with the skin


How to best transmit the electric pulse through the efficiently and without pain is a challenge in all forms of electric stimulation. The BRAID system uses standard medical electrodes used for TENS (transcutaneous electric nerve stimulation) devices that are popular with passive exercising machines and electrophysiotherapy. I settled for these electrodes because they allowed me to progress faster with my research than I was while making the electrodes by hand. The electrodes are round and have a diameter of 3.5cm and come with very steady 2mm wide nozzles for external connection.

The size and shape of the electrode has a role in the electric field that they create around the inner ear, so the electrodes are in fact the most essential element in successful GVS.

The mastoid processes are a rather difficult area of the body to apply electrodes, they are round, they are often covered by hair and the ear is too close while it must be avoided as it is more sensitive to electricity and has a lower pain threshold than other parts of the skin.

After wearing the electrodes on the stage for a session they often become less sticky and end up peeling off. We opted for surgical tape to keep them in place on the stage but this will have to be researched further in future versions of the system.

Future work


During the design and making of this system I have learned a great deal about the body and its interaction with electronics. The devices are rather bulky in their current design and there are microcontroller platforms today that have a much smaller footprints and offer a wider range of features. I estimate that the size of the device can be brought down to the footprint of a wristwatch with existing technologies and with a small redesign of the stimulation module.

The recording and playback feature has a great deal of artistic potential allowing for installations or geographically dislocated proprioceptive experiences and I would like to explore these possibilities in future works.

fig. 4 - Structure of the maculas in the inner ear

fig. 4 - Schematic of how head movement affects the maculae.

fig. 5 - location of mastoid processes

fig 6. drawing two parallel lines on the floor without vestibular stimulation

fig 7. drawing two parallel lines on the floor under the influence of vestibular stimulation

fig. 8 - Two devices of the BRAID system, with one mounted on a wig stand.