On Complex Documents

IMG_2004

Immersive VR … Google Cardboard (+ random biting cat): photo by @lynrobinson cc-by

Immersive Documents

I have written previously on the conceptual and likely practical relevance of immersive documents to the library and information science community. I have defined immersive documents as those which deliver an unreal reality to the ‘reader’. Reader, in this context is a loosely defined term, as the concept of the document is expanded to embrace the type of experience afforded by technologies such as virtual reality (VR), pervasive computing and the multisensory Internet. In this case, the reader may also be described as a viewer, a player, a user or a participant, but in writing from the perspective of LIS, the term reader seems to be an apt, all-encompassing descriptive.

This idea was sparked by my reading, some years ago, of Shuman’s scenario of the library as the ‘experience parlour’, (Shuman,1989).

Immersive documents, wherein reader engagement delivers the perception of an unreal, computer-generated world as indistinguishable from reality, do not yet exist.

This type of intangible document would, like other digital documents, exist only when its overarching computer program executes, and the associated file of coded content is read and processed. An immersive document differs from the familiar digital versions of texts, images, sound and film, (whether born digital or scanned), in that additionally, it allows for varying degrees of real-time reader behavior and interaction data to be processed along with the base content, and to influence the narrative outcome. The scope of participation or interaction could range from passive watching to full body telepresence with complete agency. The term ‘narrative’ can imply that the immersive document delivers the perception of being within a fictional novel, or a game. Whilst this is certainly one example of an immersive document, the ‘narrative’ could be a rendering of an historical event, a travel or news documentary, or a training scenario.

It is possible to regard the computer file, which contains the code for the immersive document, to be a document in its own right. It would have a physical materiality manifested in the computer storage media containing the binary codes. A sort of meta-document perhaps.

Additionally, should the specific outcome/modification from any participatory interaction be recorded, resulting in a new version of the original immersive experience, then this would constitute a related yet different document. This could in theory be played back by another participant, either as a passive, or further interactive experience.

Immersive documents comprise technology, software, and novel narratives, and developments in all three component areas will be needed for such documents to be realized, although an additional, important driving factor will be the strong desire for participatory experiences from readers.

At the time of writing, Spring 2016, we are anticipating the first wave of commercial VR head mounted displays, (HMDs), from Oculus, Samsung, HTC and Sony, which work with compatible computers and software to render stereoscopic, computer generated, virtual worlds, accompanied by sophisticated sound. These environments are compelling, and the ones I have tried deliver a realistic experience of being in a virtual world as an observer.

At this stage however, the reader is not able to fully interact with the environment or content; in programs which support telepresence, this only extends to feeling parts of the body. Doubtless, as technology advances, a fuller sense of body presence in the unreal reality will emerge, but this needs to be matched by the authoring of scripted worlds, to allow for more reader determined behavior and interaction with elements of the unreal world portrayed. [See, for example, http://motherboard.vice.com/read/tribeca-film-festival-2016-virtual-reality-film]

In support of an enhanced feeling of immersion, the HMD interface needs improvement, as at the moment it is rather clumsy and restricts the sense of full immersion, or the suspension of disbelief. Pervasive, wearable technology will undoubtedly improve to the point at which we become less/(un) aware of the interface, and we can look forward to contributions from the fields of neuroscience and psychology in reducing the friction between the reality/unreality interface. EEG headsets already allow merging of brain signals with the machine, brain-machine connection, so it will doubtless be an incremental step for this type of data recording to feed into immersive documents to simulate all five senses. [see for example, http://www.independent.co.uk/news/science/drones-brain-thoughts-controlled-bci-brain-computer-interface-brain-controlled-interface-a6996781.html]

See also, work on implants, leading to body–machine connection; cyborgs and biohacking.

Advances in multisensory transmission over the Internet, i.e. smell, taste and touch, will further enhance our ability to make the unreal, real, and at a distance.

Although fully immersive documents do not yet exist, it would be prudent for the LIS community to consider at this stage, whether the sector should play any part in the handling of these entities, and if so, in which ways. It will be easier to collect and record the documents as they emerge, if frameworks for understanding and description are already in place – thus avoiding the enormous retro-conversion efforts needed to redesign and extend current bibliographic data to enable semantic web functionality and promote discovery.

Partially Immersive Documents

Partially immersive documents do exist, and this prompts enquiry into how these can be recorded, stored, described, discovered, shared and preserved. Whilst LIS related work on these partially immersive entities is scattered amongst other disciplines, and in no way comprehensive, it is a worthwhile source of material relevant to the handling of future immersive documents, and is by its nature surely of interest to the LIS community.

Partially immersive documents may be distinguished from other analogue/physical or digital documents, because, like immersive documents, they allow for, and may even require some level of input or participation, from a data source or a reader. The distinguishing feature of these documents from other digital entities is that they are dynamic, not static.

These partially immersive documents may be divided into two categories.

Firstly, born digital entities, such as: visualizations, simulations, interactive narratives, videogames, virtual worlds, 360 digital video recordings and digital artworks. These documents all furnish the reader with varying degrees of unreal reality.

As with fully immersive documents, the level of participation within partially immersive documents can vary, from almost passive observation through to meaningful interaction – that is interaction which changes some aspect of the documentary experience.

In contrast to fully immersive documents, however, real world elements are present and noticeable, even if the reader is too ‘engrossed’ in the document to notice them.

These partially immersive document entities also exist as computer files containing content together with display or processing instructions, which require specific technological platforms on which to run. They exist only when the content data is acted upon by the software instructions. In many cases, such as with interactive narratives, there is scope for real-time data input from the reader, which generates novel content. Formats such as visualisations, simulations and digital art can all rely on other program data for input as well taking input from a human reader.

This complex, dynamic nature demands a more detailed approach to document handling than that used for digital files representing more conventional (often originally physical) types of static document, such as books, journals, manuscripts, datasets, sound, images or films, even though standard metadata for these more familiar documents may still need to be agreed, and issues of preservation in perpetuity remain.

Secondly, we need to consider partially immersive documents which are ephemeral, temporal, intangible, real world activities. Examples include theatre performances, dance performances and installation art. The level of reader participation may vary between passive reception of the content, to active engagement.

In these cases, the sense of unreality as reality is more related to suspension of reality in the mind, as the readers are at all times perceiving a real world event, even if a fantastical one. These documents need first to be recorded in order to be preserved for future access and understanding.

Augmented reality, and mixed reality events offer yet more time dependent document forms, blending physical world immersive events with the digital.

Complex Documents and the Information Communication Chain

Immersive and partially immersive documents may be thought of as ‘complex documents’, for which an interdisciplinary approach to their information communication chain journey may be beneficial, in contrast to the solely LIS focused efforts to record more usual document forms.

One area in which a significant amount or research has been done is that of preservation, and there are several disciplines which have started to consider how to describe and record complex documents within their domain.

The JISC (Joint Information Systems Committee) organized the POCOS project (Preservation of Complex Objects Symposia) in 2010, which considered complex documents as complex digital objects. To simplify the many types of objects under scrutiny, they were divided into three catagories and investigated from the perspectives of simulations and visualisations, software-based art, and gaming environments and virtual worlds. The original publications from the project are published in “Preserving Complex Digital Objects” edited by Janet Delve and David Anderson, Facet, 2014.

One of the key insights to the description of complex documents comes from consideration of the recording and preservation of dance. Firstly, we need to understand what has to be recorded. At first glance, it seems that we could perhaps merely take a video recording of the performance. But on further consideration, and indeed conversations with dancers, it becomes clear that there each performance (or dance), comprises many layers. It is likely that these layers exist for all complex documents. An initial list of things to be coded, and recorded in description and preservation systems includes:

  • 2D visual recording of the performance
  • 3D visual recording of the performance
  • coding of the movement, feelings, intention of the performers
  • sensory, (anatomical, electrical signal) recording from performers
  • mental, descriptive, narrative from performers
  • similar recordings from the creator, director
  • psychological, physical reaction from participants
  • overall impact of the performance

Perhaps the main challenge to handling complex documents within the information communication chain comes from reader participation.

Participatory behavior can take many forms, from allowing software to read facial expressions, or to measure pulse or heart rate, to allowing full interaction with objects, characters or avatars within the simulated environment. This data is complex to record and process with respect to the base document, but is also complex to add to document description. In some cases, there is more than one participant, adding to the complexity. For the purposes of recording and preservation, there is the question of what is being recorded, and how authentic is the replay?

In some games for example, there are hundreds of participants. Likewise for immersive, participatory theatre. Once we consider participation, we are faced with an additional layer to the recording, description and sharing of the document – do we record the viewpoint/experience of the participant? Can we? There are thus dual (possibly multiple) reader modes; that of a passive observer, a first time reader who is interacting with the document, or that of previous readers, engaging with something previously experienced..

It is perhaps surprising that the question ‘what is a document?’ is still unclear, five and a half thousand years after records began. Humanity nonetheless, still endeavors to record and preserve more and more layers of the human condition.

 

References

Shuman BA (1989). The Library of the future. Alternative scenarios for the information profession. Englewood CO: Libraries Unlimited.

LR update 2/05/16

 

On The Pogles: What makes an immersive document?

Here is an episode of The Pogles, a children’s television program from the 1960s. The film is in stop motion, made by the wonderful Oliver Postgate.

I write nowadays about immersive documents emerging from exotic technology, new narrative writing methods, and a populace madly keen to escape reality via participation in unreal realities. Except that, just sometimes, we can be transported into the unreal realm by far simpler mechanisms. I have written previously about immersive stories (see here), and in this small piece of writing I invite you to spend 9 minutes or so telling yourself that Pogles and magic plants don’t exist!

Note also:

1) In the 60s, alcoholic beverages are a *good thing*

2) In the 60s, there is no email and so plenty of time for adventure

21st Century Library and Information Science

Time .. always changing things ..

Time .. always changing things ..

Some thoughts based on my recent presentation to the INFODAYS14 conference held at Charles University, Prague, Czech Republic 5-7th November 2014.

The future of library and information science (LIS), is inextricably linked to the future of the document. Leaving aside for the moment, the question of exactly “what is a document?” this comes as no surprise to those of us working in this area, as we know that civilization owes its existence to recorded knowledge. For the time being also, let us allow ourselves to consider knowledge and information to be interchangeable terms, and within our LIS discipline, we will assume that for the purposes of communication, such knowledge or information must be instantiated as a document of some kind.

We can further understand that documents contain, and therefore allow access to, ‘formal’ information – i.e. something that is published, and therefore subject to the processes of the information communication chain, [Popper’s World III, instantiated in World I, physical objects]. This is in contrast to ‘informal’ information, which remains inside someone’s head – although developments in telepathic communication are starting to enter the research literature, we are still some way away from being able to intercept and understanding the thought processes of another being, [Popper’s World II].

I am often called on to comment on the nature of library and information science. To reiterate, here at City University London, we consider the discipline of library and information science to be the study of documents, on their journey through the information communication chain.

The realm of library and information science then, is the realm of the document. We, as researchers and practitioners within this field, are concerned with the activities surrounding the creation, dissemination, management, organisation and retrieval, and use of documents. We study these processes of documentation through the lens of Hjørland’s concept of domain analysis, invoking study and praxis within facets such as knowledge organisation, information retrieval, document preservation, historical studies, and research into information behaviour.

The changes in this chain of events, are driven largely by technology, although factors such as economics, politics and social tastes are all able to impact on the business of recorded knowledge.

Let us take a look at some of the developments that can be seen to be influencing the nature and definition of documents, as 2014 draws to a close.

Information Theory

A quick scan of the literature on the definition of information, reveals the troubled history of the concept central to our discipline. To-date, there is no single, satisfactory explanation of what information actually is. There are many attempts at definition, and indeed theories, both within the field of LIS and within other, seemingly unrelated disciplines. Resurgence in interest in information communications theory, can be seen to have heralded interest in information physics, philosophy of information and information biology. To some, the concept of information in these fields remains unconnected to the social discipline of library and information science, but to others, there is an interest in looking for connections and a possible theory of unification. See Bawden and Robinson papers below, for further reading in this latter area.

Data Science

The jump to prominence of data science and related areas (big data, data analytics, data visualisation) can be seen in the number of recent university courses being introduced (we have introduced one this year at City University), and reflected in the required skills listed in many job advertisements. One of the top skills sets required by employers across the sectors is the ability to collect, analyse and interpret data. Data handling and communication are now sitting alongside the more traditional ‘verbal and written communication skills’ that professional level work demands. These skills are becoming essential to practitioners within library and information science as e-science and the digital humanities pour more and more data sets into our sphere of influence. These digital data files are one example of the new forms of document that require the attention and understanding of members of our discipline. The move to open data, and the expectation that data will be published alongside findings are changing the way library and information professionals support scholarly communication. Indeed the move towards digital scholarship heralds a new era for partnerships between librarians and researchers.

Digital Humanities

The mass digitization of literature, poetry, art and music has led to an increase in materials and methods available for humanities based studies. There is a movement towards situating digital humanities research within the library and information environment, which seems to many, to be its natural home. Again, as with e-science, the availability of large data sets and multi-media files is fuelling new growth areas for understanding patterns and trends (text mining), and for facilitating the final convergence of the GLAM sector, where digital renderings of text, image, sounds or even objects bring the previously separate collection disciplines into a melting pot of new services and interpretations. We are witnessing new roles redefining library and information science as a producer of new content, understanding and insight, supporter of new forms of scholarship, and a leader in scholarly communications.

Publishing

Library and information science has always had an intimate relationship with the processes of publishing and dissemination. Changes in both scholarly and trade publishing are well documented, driven by the open access movement and the demand for new models of consumption respectively. The rapid growth of mobile devices and social media has revolutionised what it means to be an author and what it means to be a disseminator or a reader. It is probably fair to say that anyone with access to technology (not everyone) can be both an author and a publisher. New mechanisms for content creation (image/media capture devices, writing for transmedia) allow new forms of documents (interactive narratives) to flourish, and we are seeing a move towards content marketing, an increase in the use of images or video over text, and in data mashups. New tools to help us understand the reach and potential impact of new publishing formats, referred to as altmetrics, are entering the armoury of library and information science alongside existing bibliometric and informetric analyses. What it means to publish is changing alongside the development of the document.

Computer Science/Technology

Technological advances undoubtedly drive the most significant changes in the form and nature of documents.

I have written previously in this blog, that developments in pervasive computing, multisensory network technologies and participatory human computer interfaces will allow new forms of documents to emerge, specifically ‘immersive’ documents, where unreality can be perceived as reality. News of current developments in virtual reality headsets and roomscape projection abounds, and consumer versions of games, narratives and training scenarios appear to be just around the corner timewise, rather than siting themselves somewhere in the mid to long term future. Before we arrive at the availability of completely immersive documents, we will see a range of lesser, participatory experiences, such as the interactive, transmedia narratives mentioned in the previous section. In these narratives, the story reaches out beyond the imaginary world, into the reality of the reader, with texts, phone calls and connections, seemingly coming from characters within the plot. The way the narrative plays out can be influenced by the reader, as can the ending.

The blurring of boundaries between a game, a learning experience or pure fiction with this type of document is evident. There will also be ethical implications with regard to how these documents are used.

Implications for Library and Information Science

As documents evolve, so then will the scope and processes of what we understand as library and information science. New forms of document will require extensions and adaptations to our current tools for knowledge organisation, new information architectures and new understandings of human information behaviours. Most interestingly perhaps, for the LIS profession, will be the need to engage with and promote ‘immersive literacy’, possibly in a similar way to which Gilster suggested for digital literacy less than two decades ago.

Further Reading

Bawden D and Robinson L (2013). “Deep down things”: in what ways is information physical, and why does it matter for LIS? Information Research 18(3), paper C03 [online], available at http://InformationR.net/ir/18-3/colis/paperC03.html

Gilster P (1997). Digital Literacy. New York NY: Wiley, New York.

Hjørland B (2002). Domain Analysis in Information Science: Eleven approaches – traditional as well as innovative. Journal of Documentation, vol 58(4), 422-462.

Robinson L (2009). Information Science: the communication chain and domain analysis. Journal of Documentation, vol 65(4), 578-591.

Robinson L and Bawden D (2013). Mind the gap: transitions between concepts of information in varied domains. In: Theories of information, communication and knowledge. A multidisciplinary approach. Eds. Ibekwe-SanJuan F and Dousa T. Springer.

Robinson L (2014a). Multisensory, Pervasive, Immersive: towards a new generation of documents. Journal of the Association for Information Science and Technology, in press.

Robinson L (2014b). Immersive information behaviour; using the documents of the future. New Library World, in press.