Print as remediation for the multimedia? Elasticity of multimodal narratives

Introduction

In this article, I consider remediation from the opposite point of view we usually adopt when dealing with the subject. The term is mostly used to talk about new media, as Bolter and Grusin’s book title indicates: how visual content is refashioned and/or repurposed in digital literature; how fixed and moving images, whether they are extracted from an existing context or created ex nihilo, are used in digital works that question the circulation of cultural objects. However, the objects I want to focus on here are not digital but printed. They constitute in part what Alison Gibbons calls “multimodal literature,” which she defines as follows:

Multimodal literature as a genre is not uniform, but rather exists on a spectrum, from minimal to extensive in the level of incorporation of multimodality. Considering multimodal literature from a holistic perspective, some of the formal features these works tend to contain are: varied typography, unusual textual layouts and page design including the concrete arrangement of text for visual purposes, the inclusion of images (illustrative, diagrammatic, photographic) and facsimiles of documents; multimodal literature may play with the size, shape, and design of the codex, using cut-outs/die cuts or pop-ups, and offering the reader throw-outs or flip book sections; multimodal literature, perhaps partly because of the striking impact of its visuality for readers, often pushes at its own ontological boundaries, whether in the form of metafictive writing, footnotes and self-interrogative critical voices, or through ontological masquerade in itself. (“Multimodal Literature” 420)

Gibbons further writes that “relationships can be found with other literary genres, the most obvious being children’s picture books […] or graphic narratives” (420), which allows me to expand her definition and include those media under the term “multimodal narratives.” To sum up, multimodal narratives question the codex as form and medium, include a strong text-image interaction (that can sometimes result in the fusion of these two elements) and play with the materiality of their format.

I would like to start with Gibbons’s definition and consider it from a Bakhtinian point of view. In The Dialogic Imagination, Bakhtin writes about the novel:

In principle, any genre could be included in [its] construction […] and in fact it is difficult to find any genres that have not at some point been incorporated into a novel by someone. Such incorporated genres usually preserve within the novel their own structural integrity and independence, as well as their own linguistic and stylistic peculiarities. (320-321)

The French translation of this extract uses the word “elasticity,” which I think defines well what I am trying to convey. Bakhtin also writes about “a special group of genres” that can “determine the form of the novel as a whole (the novel-confession, the novel-diary, the novel-in-letters, etc.),” genres that, when incorporated into the novel, deny it “any primary means for verbally appropriating reality” and “bring into it their own languages” (321). Bakhtin further discusses how poetry in verse can be inserted into a prose text, but I am more interested in how his observation about the elasticity of the novel could apply to multimodal narratives and the way they remediate and/or assimilate other cultural objects in the making of their narrative. Indeed, multimodal narratives, in their incorporation of other modes than text, can be considered as remediating the media associated with these modes. To prove this assumption, I am first going to analyze how print remediates cinematographic techniques, before considering the remediation of other media and senses as well as the aesthetic devices multimodal narratives can use by taking advantage of their printed nature.1

Moving pictures

A famous example of how cinematographic techniques are remediated within multimodal narratives is House of Leaves, a novel published in 2000 by Mark Z. Danielewski. Its plot is so complex that I will only mention how it partly deals with the exploration of the labyrinthine underground of a house by its owner, documentary maker Will Navidson. As N. Katherine Hayles notes:

The inscription technologies [featured in House of Leaves] include film, video, photography, tattoos, typewriters, telegraphy, handwriting, and digital computers. The inscription surfaces are no less varied, as Johnny Truant observes about Zampanò’s notes, which include writings on ‘old napkins, the tattered edges of an envelope, once even on the back of a postage stamp; everything and anything but empty; each fragment completely covered with the creep of years and years of ink pronouncements; layered, crossed out, amended; handwritten, typed; legible, illegible; impenetrable, lucid; torn, stained, scotch taped; some bits crisp and clean, others faded, burnt or folded and refolded so many times the creases have obliterated whole passages of god knows what—sense? truth? deceit?’ (xvii)” (780-81)

This is, as Hayles writes further, a “frenzy of remediation,” an attempt “to eat all the other media” (781), akin to the consumption of the romance by the novel Bakhtin alludes to. There are many instances of such attempts within the novel. For example, from page 294 to 296, a word is fragmented into three segments to better reproduce the action described; there are many other moments in the novel where “typography serves to produce cinematic effects of temporal acceleration and deceleration” (Hansen 617), as in chapter 10. Those are cinematic but also kinetic effects which create the illusion that the time of reading and the time of the action coincide; Hayles, when writing about those devices, compares the eye movement of the reader on the page to how spectators would look at it in a movie theater (796). A final example would be how a short film’s tracking shot is described within the text, “textually translated by the presence of a single and long sentence. Each segment of the sentence between commas could correspond to one picture, and the sentence would thus constitute one shot” (Guilet, House of Leaves 50).

Danielewski also uses editing techniques in the novel, particularly in chapter 8, where they are either directly mentioned within Zampanò’s commentary of Navidson’s documentary or correspond to effects caused by the text’s structure. Chapter 8 begins with the sentence “Billy Reston glides into frame” (97), making the character appear both in the film but also in the text, before Zampanò describes Navidson’s decision to keep the camera on Reston and himself to reinforce the spectator’s—and the reader’s—identification with these characters. A paragraph made of concise sentences describes the tension building between them: “Jump cuts increase. People stop speaking to each other. A single shot never includes more than one person” (101), alternating between descriptions of the film’s editing and the characters’ actions and feelings. Finally, at the end of the chapter, Danielewski anticipates the critical discourse by having Zampanò quote a fictional scholar, Tasha K. Wheelston, who comments on a visual S.O.S. in Navidson’s documentary, when three short shots and three long shots alternate. This call for help is also present within the story (the characters hear short and long bangs on a wall) and in the novel’s layout, since the chapter’s text is divided by Morse code signals into three short sections followed by three long sections.2

A final example of how Danielewski uses editing techniques is in the sequence leading from chapter 18 to chapter 22. Chapter 18 ends on an unfinished sentence, without a period: “[Karen] finds out that the video tape shows” (417). The next chapter begins with the sentence: “‘Nothing of consequence’ was how Navidson described the quality of the film” (418); the reader is made to think that “nothing” is precisely what Karen sees (or more accurately does not see). But the real end of the sentence is at the beginning of chapter 22: “Nothing more now than the mere dark. The tape is blank” (522). The whole, reconstructed sentence therefore reads as follows: “[Karen] finds out that the video tape shows nothing more now than the mere dark. The tape is blank.” Chapters 19 to 21, inserted between the beginning and the end of this sentence, thus resemble inverted reels, or a deliberate continuity error.

Examples of effects meant to remediate moving pictures on the printed page abound, both in House of Leaves and other multimodal narratives, but none is as potent as the use of literally moving images or text through the use of flipbooks within the narration. Flipbooks were invented before cinema proper in 1868, but they remain closely linked to the medium as well as to the book format, which explains why they are a perfect tool for remediation. A sequence in chapter 10 of House of Leaves could be considered as such: the sentence “All those doors behind the man are slamming shut, one after another after another, which still does not prevent the figure from firing” is spread out across 9 pages. Another example would be the way paragraph sizes diminish and page numbers revolve in Danielewski’s second novel, Only Revolutions (2007). The most explicit cinematic flipbook, however is found in Steven Hall’s The Raw Shark Texts (2007), in which Eric, the protagonist, is attacked by a conceptual shark. The shark, represented by calligrammatic blocks of text, is first stopped from attacking by physical surfaces, such as a tiled floor or a TV screen, but it eventually manages to come through both the story and the book’s pages, moving across several pages towards both Eric and the reader. The homage to Jaws is explicit, but I think the sequence also alludes to L’Arrivée d’un train en gare de La Ciotat, the Lumière brothers’ movie which allegedly scared some spectators out of the projection room. Furthermore, just like the shark slowly approaches in Steven Spielberg’s film, the reader must be careful to not flip too fast and skip pages since the flipbook is not at the end of the novel: the device seems to incite one to slowly turn the pages.

Though I have only mentioned text-based techniques so far, there are of course image-based flipbooks, most famously (and most recently) at the end of Jonathan Safran Foer’s Extremely Loud & Incredibly Close (2005), where 15 still frames of a man jumping off the Twin Towers are rearranged to give the impression he is flying upwards. Far from a trivialization of the terrorist attacks and the trauma they caused, the flipbook is a link to the novel’s main themes, such as the need to protect loved ones and the impossibility of making sense of violent images in contemporary visual culture. Oskar, the story’s main character, wishes his father had not died in the attacks and that saving him was as easy as ripping out pages and reversing their order. The novel ends on a series of inversions: of the frames, of the character-turned-author, and of the reader-turned-actor in the sense that s/he turns the pages as Oskar does. It is also a commentary on how 9/11 changed our perception of time, as Alison Gibbons writes: “In light of the events of September 11, 2001, time cannot be understood as an ongoing linear process since the future is haunted by the past and the past haunted by imagined temporal possibilities or ‘counter-narratives’” (Multimodality 162). Though not an explicit reference to cinema, the use of a flipbook in Safran Foer’s novel could also be considered as a commentary on the news media’s visual treatment of 9/11, both in its repetitive and dramatic aspects.

Finally, flipbooks are also and obviously used in the comics medium, for instance by Chris Ware who references cinema repeatedly throughout his work: there is a zoopraxiscope and a zoetrope in Jimmy Corrigan (2003), a detachable flipbook in Acme Novelty Library issue 7, a “movie machine” in issue 3 of the same series and a “kine-comics” page in Quimby the Mouse. All these references to the prehistory of cinema show Ware’s desire to create comics that are read quickly—since details from one image to the next don’t vary much in a flipbook—but also in which the time of action is slowed down by the repetition of almost similar panels. To place comics at the end of a cinematographic lineage is to define it primarily through movement (of the eye moving from one panel to the next and of the hand turning pages). A similar intent is at the heart of the flipbooks I’ve mentioned: they are not only about literally moving the story forward and synchronizing the time of action and the time of reading, but also about making the reader an active agent of the narration. Turning pages remains a mechanical act, but in the primary meaning of the word: it sets the story going, just like turning on a projector would.

Sounds and smells

The previous examples show that print can easily remediate movement, just as it can remediate sound, and even smell, through devices relying once again on the intervention of the reader and on touch. Most of those devices are found in children’s books, which heavily rely on interactivity so as to engage their young audience, even though Barbara Wall astutely remarks that children’s books are also for adults, who sometimes read the books instead of the child, or with him/her (2).

Many examples of “Sound Books” are found in children’s literature: the push of a button or the turning of a dial produces noise, either electronically (taking a page from digital literature) or mechanically, as is the case in Jan Pieńkowski’s famous Haunted House, published in 1979 and which features, amongst other interactive aspects, a pop-up of a saw going back and forth through wood as the reader opens and closes the book.

Smell can be remediated as easily as sound, though its use is much rarer and not as popular as it was in the 1980s, when Little Golden Books produced a series of “Scratch and Sniff” books, where surfaces had to be scratched by the reader to release chemically-engineered smells of various origin. The “Scratch and Sniff” device also seems to be mostly limited to children’s literature, although I have also found it used in comics (DC Comics produced one in 2015 and Ian Hague evokes “early printings of the manga series Antique Bakery” which “featured scratch and sniff covers” [“Beyond the Visual” 102]) and in cooking manuals. It is true that smell has never been a prominent sense in reading, in contrast to sight, touch and hearing3; on the other hand, many scholars mention the characteristic smell (or smells, to be more accurate) of paper, as Matija Strlič indicates in an interview: “We know that books produced before approximately 1850 have a different smell to those produced between 1850 and 1990, […] because late 19th- and most 20th-century printing was dominated by acid sizing” (Armitstead, n.p.). Smell thus characterizes not only the age of a book but also the nostalgia sometimes associated with the print medium, which is not so much remediated as created by it (see also Hague, Comics and the Senses 123-149).

What print does that screens can’t

Interestingly, digital literature often tries to imitate the appearance, texture and sound of print; most of our digital reading practices, in fact, borrow from the skeuomorphic image of the book (one could incidentally argue this is what House of Leaves is doing with cinema). The term comes from the Greek ‘skeuos’ (costume, ornament, decoration) and ‘morphos’ (form, external appearance); in the design field, skeuomorphism is used to describe ‘a visual element whose form isn’t directly related to its function, but which reproduces in an ornamental way an element that was necessary in the original object; for instance interface elements reproducing physical objects’ (Graphéine n.p.). On an iOS-powered device, one can find for instance the notebook icon used for the ‘Notes’ app, the clapperboard icon for the ‘Videos’ app or the open book icon for the ‘iBooks’ app. That most of the navigation interfaces of epub and PDF files imitated for a long time the act of reading a printed page, with the animation of the turning of a page (combined with a sound also imitating paper) is a sign of the dependence of the digital on paper (though it is less the case nowadays).

Although the common idea is to associate remediation with digital media, there are lots of things that printed books do that screens can’t, to paraphrase Seymour Chatman’s seminal article, such as manipulating the page’s materiality. To quote Anaïs Guilet, it is conventionally considered that “the book is stable, the linked screen is fluid” (Pour une littérature cyborg 143), but books can be modified in ways that screens can’t. “A page is not just ‘an area’,” writes Alan Trotter; “it is sculptural, an object that exists in space. Therefore, you can cut into it; therefore you can bind it to others like it in different ways or not at all” (13). Observing two different devices relying on the materiality of the page—by piercing it or folding it—will allow me to reestablish print and digital narratives as different ways of exploring a narrative’s multimodality, instead of considering the two media hierarchically.

Piercing the page, reaffirming its tridimensionality by hollowing its surface, is not a recent device, though it has certainly been made easier by the evolution of printing technology. It was most probably difficult to die-cut a few pages in every copy of B.S. Johnson’s Albert Angelo (published in 1964), whereas it is possible today to print relatively easily a book like Jonathan Safran Foer’s Tree of Codes (2010) in which every page is excised.

Much has been written about Albert Angelo and Johnson’s use of die-cut pages in the last part of the novel. Before the hole (which, at the bottom of the page, lets one see three lines printed four pages ahead) is a scene in which Albert tells his friend Terry about the violence in the school where he teaches and his attempts at defusing this violence. The lines we see through the hole—“struggle to take back his knife, and inflicted on him a mortal wound…”—seem to prove he fails and maybe even gets killed by one of his students. When the reader reaches the page revealed, it turns out to be an extract from an article about Christopher Marlowe’s death, something seemingly unrelated to the general narrative.

There are three ways of looking at this device. Jonathan Coe sees in this short-circuiting of the reader’s expectations “a cryptic reference to Johnson’s identification with Marlowe (who was killed at the age of twenty-nine in a tavern brawl) and belief in his own imminent death” (125), as well as, and more importantly, the denial or dissimulation of Johnson’s “homoerotic feelings” for a friend of his (445).4 For Vanessa Guignery, devices like the hole in Albert Angelo both emphasize the status of the book as an artifact and fulfill a mimetic function (207): the hole mimics the mortal wound inflicted by the knife on the victim’s skin (209), and wounds the stability of the book itself. Finally, Patricia Waugh argues it forces the reader to think about the way suspense is conventionally constructed in literature (96). The hole is not unequivocal: it is at the same time an absence and an injury, a revelation and a dissimulation, a metacomment and an ironic false clue.

Absence is also at the heart of L’Origine, Marc-Antoine Mathieu’s bande dessinée published in 1990 and which, like Albert Angelo, became famous for featuring a hole in lieu of a panel in one of its pages (an “anti-panel,” as one character calls it) which allows both readers and characters to see into the future. L’Origine is built around the motif of metatextuality: its story begins with a reference to similar panels in Winsor McCay’s Little Nemo in Slumberland, implicitly defining Julius’s world as wavering between dreams and comics. The book is also full of grid-like patterns evoking the layout of a comics page: the plane on which Julius walks in his dream, his pajamas, bathroom tiles, windows, floor tiles, and so on. The motif is almost everywhere in this story, in which Julius receives comic pages foretelling his own future. He eventually discovers that he is living inside a comic book, a two-dimensional world which is included in a tridimensional world (ours). In Julius’s world, according to a scientist he meets, anti-panels exist and allow one to travel back and forth in time. Indeed, if readers see into the future of the story when they encounter the hole, by turning the page and encountering it again, they will see into the past, as the story is keen on reminding us. To quote Sylvain Lesage: “By hollowing out the comics page, Marc-Antoine Mathieu brings out the true nature of the founding narrative mechanism of comics: in them, time is only space” (24). The holes in Albert Angelo and L’Origine are thus both similar (they make us perceive other parts or other times of the story) and different in their effect.

Books with holes are extremely frequent in children’s literature, as if the pleasure of stumbling upon haptic devices was reserved to kids. However, as Éléonore Hamaide-Jager observes, “they all work on the same register, often in a somewhat lazy way, without thinking about the effect on the reader or trying to build a story in relation with the chosen form” (122). Exceptions exist, however, such as Øyvind Torseter’s Hullet (“The Hole”), published in 2012, in which a hole traverses the whole book from cover to cover. The protagonist of the story moves into a new apartment and quickly realizes there is a hole in it, and spends the rest of the book trying to get rid of it. Just like in L’Origine, this hole is both a rupture in time and space; indeed, as Hamaide-Jager writes, “the book’s great idea is to make us think the hole moves thanks to the character’s […] movements, whereas it is materially fixed and central. […] The hole seems to move by itself thanks to moving graphic points of view which give the illusion it is animated by its own will […] it is not a circle on the page but a place, inscribed on it” (124). The hole thus has a function similar to the die-cut pages in Albert Angelo by being both a presence and an absence, both putting forward the presence of the page as site and expanding beyond its conventional bidimensionality.

The manipulation of the page through its folding, though less common and possibly more experimental, represents a different way to forefront the materiality of the book and explore its tridimensionality. As with the previous examples, the aim is often to move from surface to volume, to expand both the book and its narrative beyond what was seen before opening the covers. Examples abound in children’s literature, from books with flaps to accordion books; the latter is also, if not common, frequent in comics, the most recent example being Joe Sacco’s The Great War (which, it can be argued, is not exactly a comic, though it was marketed as such). In multimodal narratives, such a device can be prominently found in Adam Thirlwell’s Kapow! (2012). Kapow!’s narrator is a London writer who begins his story by recounting a meeting with a taxi driver, who tells him about his friends in Egypt and the revolution that is going on there (the story roughly takes place in the beginning of 2011). The narrator decides he is going to invent a love story between the people the taxi driver is talking about, but his narration is frequently interrupted by considerations on the structure of his discourse and his legitimacy to talk about a country he knows nothing about. Integrity is indeed something he is very concerned about: “My idea of integrity,” he says, “meant that you had to follow every thought as far as you could, into all the dead ends” (18). This conception of narration leads to numerous footnotes and asides that grow exponentially until it is necessary to expand the page itself to make enough room for everything the narrator wants to say. As the story goes on, his invented characters begin to make decisions that he does not control; as Hugo Ferraz Gomes writes, “their expansion is emulated by the novel’s enfolding and spreading out pages, where deviations of the main narrative path are offered” (68). Ideally, Kapow! would go on and expand indefinitely as it explores every deviation possible, but since it is constricted by the spatial limitations of its printed form, it must and does end, without any real resolution.

By putting the materiality of its pages in the foreground, multimodal narratives are thus able not only to remediate a lot of different media but also to invent devices which cannot be digitally remediated or only through an important aesthetic and/or narrative loss: digital pop-up books exist, for instance, but what they make up for in flourish, they lack in interactivity, which is ironic when comparing the two media’s (supposed) capacities in this aspect.

Conclusion

I wish to reiterate that my argument is not that print is superior to the digital or that it has developed against or in spite of it. On the contrary, works like Danielewski’s novels or Steve Tomasula’s VAS (2002) are “rendered anew by the digital,” to quote Kent Aardse (49); digital tools make it easier for them to explore the concept of remediation and the materiality of the page. Something should also be said about the financial benefits of multimodal narratives compared to digital literature, the latter remaining today a niche market for both editors and authors.

I mentioned in my introduction Bakhtin’s remark about “a special group of genres” that, when incorporated into the novel, deny it “any primary means for verbally appropriating reality” and “bring into it their own languages” (“Discourse” 321). It seems to me that is what multimodal narratives do by remediating movement, playing with the surface of the page as if it were a screen; by remediating sound and smell, enhancing the conventional reading experience; and by foregrounding their materiality, revealing that the page is not only a surface but also has potential depth. Multimodal narratives become elastic when they welcome in their body other media and other languages that go beyond the verbal and redefine not only what we can do with a printed work of fiction, but also how we should consider it.