‘The Memory Tower’
Following an introduction, the five key components of the virtual installation are discussed in some detail with the aid of animations. For more information about this installation, including an in-depth discussion of the science behind it, please visit the ‘ISIS - Tech Talk’ link in the right side bar to download a video of a public lecture I recently gave concerning this project at the John Hope Franklin Centre, Duke University.
Memory Consolidation
Our memories are a record both of our daily experiences and our histories across generations. The processes by which new memories are consolidated from a labile form (i.e. vulnerable to disruption) to a more permanent state are still being actively researched within the field of neuroscience. The ‘Memory Tower’ is an artwork exploring the neural mechanisms behind the initial encoding and eventual consolidation during sleep of memories in the brain. It is being developed as a fully immersive virtual reality installation, with the aim of presenting current scientific understanding in a more intuitive and explorative way.
In one model of memory consolidation, perceptual, cognitive and motor information arising through waking experience is initially encoded in parallel across many cortical regions including the neocortex and the hippocampus. With regards the content of these memories, the hippocampus is believed to be important for the processing of episodic (event based) memories, semantic memories (i.e. abstractions of information from episodes) and spatial memories. More generally, therefore, it serves to link events across space and time.
Initially, memory encoding within the neocortex is thought to be distributed across many cortical modules, the hippocampus serving to bind this information into a coherent memory trace. During memory consolidation, successive reactivation of memory traces within the hippocampal-neocortical circuitry will then strengthen the connectivity between these cortical modules, stabilizing the new memories. These rounds of reactivation may also function to integrate new memories with older, long-term memories that coexist in the same neural networks.
Thus, this artwork explores the consolidation of a new memory within the rich context of earlier encoded experience. The use of historical, architectural forms embedded within a complex cityscape encompasses the episodic/semantic and spatial facets of our daily experiences, as well as serving to reflect on a sense of collective human history. When explored as a virtual environment, the waking counterparts of the cognitive processes built into the form and content of the work will also be those engaged in the act of exploration
Part 1|5 Slow Wave Sleep
Sleep is believed to play a vital role in the process of memory consolidation, with different roles ascribed to Rapid Eye Movement (REM) sleep and Slow Wave Sleep (SWS). A key feature of SWS is the presence of a cortical oscillatory pattern termed the ‘slow wave’, as seen in Human EEG sleep recordings. This slower-than 1Hz oscillation is thought to drive the memory consolidation process by synchronizing the firing of neurons in key brain areas, such as the thalamus, hippocampus and the neocortex. This slow oscillatory cortical pattern observed during SWS is represented in the three outer layers of the virtual cityscape, each representing a different period of SWS. As such network states arise through the summed firing activity of populations of cells, including those that carry information in their neuronal firing patterns, so these oscillatory patterns are described in the same form as the memory components themselves, i.e. spatially discrete, structures.
Part 2|5 Spindle and Ripple Oscillations
During this slow wave activity, a short lived and higher frequency oscillation in neural firing can be observed in the neocortex, termed a spindle (10-16 Hz). This activity is driven from the thalamus and is synchronized with the depolarizing phase of the slow wave in the neocortex, when neocortical cells are most excitable.
During this time, there is also an increase in hippocampal excitability. Bursts in hippocampal cell firing (link to spikes) are temporally correlated with these cortical spindles. At up to 200 Hz in frequency, they constitute a high-speed reactivation of packets of information encoded by the hippocampus during waking experience. This replay feeds back to the neocortex driving memory consolidation.
Part 3|5 Central Tower
At the center of the installation is a memory undergoing consolidation, taking the form of a highly ornate, baroque tower. As it is not yet fully consolidated, it remains heavily fragmented. Thus, each ripple event, originating in the hippocampus, is bound only to a component of the tower, the tower’s complete form only being perceived when the total activity of the hippocampal-neocortical network over time is taken into account (seen in the total ripple activity of the installation). Over time, the co-activity of neocortical neurons participating in the spindle event with hippocampal ripple activity undergo neural plastic changes that strengthen their connectivity, binding the cortical representations together to form a complete, stable memory trace. With time, therefore, the central tower would become bound into a coherent, unified form
Part 4|5 Long Term Memories
The consolidation of this representative memory, however, does not take place in isolation of previous encoded experience; older, related memories also reside within the neocortex. Like the central tower, these related memories also take on an architectural form, each sharing with it some unique aesthetic or historical connection. By drawing from over two thousand years of classically-derived architecture, the relatedness (i.e. the semantic content) of these memories is made explicit in addition to the episodic (historical) and spatial nature of the work. These structures exist in a greater state of completeness than the central tower, having already undergone some part of the consolidation process. By varying the forms of these previously consolidated memories, different types of memory process are modeled. Towers, for example, may represent episodic-like events (information bound together as an episode) whilst horizontally orientated buildings capture events occurring in time (both procedural memories and memories capturing sequences of episodes in time). The cityscape thus formed, provides a rich mnemonic context into which the central tower is constructed, one that renders older memories labile and drives new forms of connectivity between them
Part 5|5 REM sleep
Rapid Eye Movement (REM) sleep also forms an important part of the installation. During REM sleep, a characteristic and sustained 6-10 Hz “theta-band” oscillatory rhythm is observed in the hippocampus and other cortical areas, arising from synchronized neuronal activity. A similar theta pattern is also observed during waking activity, and it is the unique firing patterns of neurons in these brain regions relative to this ongoing oscillation that encode many aspects of our waking experiences.
Unlike the rapid and brief reactivation of ‘packets of information’ during ripple activity in the hippocampus, sequences of behavioral states encoded during waking theta activity are reactivated during REM sleep, reverberating through the hippocampal-neocortical circuitry. This replay may facilitate the integration and linking of related memories. Indeed, research suggests that our external world is endogenously simulated during REM sleep, the brain functioning as an innate virtual reality generator complete with in-built predictions of external space and time through which our waking experiences are organized. Thus, in the installation, older long-term cortical memories are linked to the ‘new memory’ through a framework of structures representing REM sleep and the role played by the theta oscillation as a time frame against which information is initially encoded and then replayed during sleep.
Variations in theta rhythm frequency may play a role in linking (phase locking) different cell ensembles between the hippocampus and neocortex, uniquely ‘binding’ specific populations together. Thus, in the installation, different memory structures are embedded within a unique frequency/form of theta network activity.
Implementation in the DiVE
The ’Duke immersive Virtual Environment’ (DiVE) at Duke University (http://vis.duke.edu/dive/overview) is a three meter cubed, six-sided virtual reality theater. By projecting stereoscopic images onto each surface of the DiVE, stereo glasses can be used to generate depth perception. A hand held “wand” can then be used to control navigation through the environment and to allow interaction with virtual objects.
The DiVE provides an opportunity to build extensive virtual environments. By constructing ‘The Memory Tower’ in the DiVE as a virtual environment, strong parallels can be drawn between the mode of experience and the nature of the phenomena that the work describes.
Firstly, the notion of the REM sleep state as a ‘virtual reality generator’ though which our waking experiences can be organized, and related scraps of memory bound together, can become an experiential reality. Not only can the idea of memory content be explored to unparalleled depths (made possible through the virtual scale of the work), but also the act of exploration and information assimilation within the environment parallels processes active in the REM sleep state. Indeed, the waking counterparts of the cognitive processes built into the form and content of the work will be those engaged in the act of exploration. Over time, as a ‘cognitive spatial map’ develops, the observer will more readily navigate the environment and understand how the individual pieces ‘fit together’.
At regular intervals, the user will encounter ‘information spheres’ within the environment, at which the press of a button will impose text onto the four walls of the DiVE Theater. These texts will describe different facets of memory encoding, consolidation and retrieval from disciplines such as the sciences, literature/poetry and architectural theory. These points of contact are designed to inform the observer and simulate a reappraisal of the virtual environment in which they are exploring.
The following images are from the Memory Tower as it will appear in the DiVE. To enlarge the images, please click the icon at the far right hand side of the slideshow tool bar.
Summary
Neuroscientists are gradually gaining insight into the multifaceted nature of memory consolidation, the importance of different brain states and the role oscillatory rhythms may play in the encoding and replay of waking experience. Through this artwork/installation, I hope to capture something of this remarkable nature of memory formation, as seen through the eyes of neuroscience. The building into ‘The Memory Tower’ of the rich content and context of memories is a deliberate move away from the [necessarily] reductionist approach adopted within research practices in the study of memory processes. By using familiar markers of memory and everyday experience in the description of biological processes underlying memory consolidation, the hope is to find more intuitive methods for communicating knowledge derived from the biological sciences to a wider, non-specialist, audience.