Rubbing as fieldwork: Workshop on site writing with technologies of captures, with MELT, Shauna Janssen and the Speculative Realities Lab at Kunstgewerbemuseum

Last week, in collaboration with the SpecLab, I had the privilege of inviting MELT (Ren Loren Britton + Iz Paehr) and Shauna Janssen from Concordia University for a three-day workshop/gathering at the Kunstgewerbemuseum, with Claudia Banz as our host. The vibrant group of participants and the museum's cosy atmosphere created a splendid contrast against its spacious and brutalist architecture.
Ren and Iz led the morning sessions, while Shauna unfortunately had to remain at home. However, her insights on “Site Writing” continued to resonate with us throughout the workshop. In the afternoons, the SpecLab team presented multiple resources related to our ongoing research on brain data navigation at Charité. The focal point of this presentation was the early-stage prototype itself, which can be seen in the image above, as neuroscientist Melina Engelhardt interacts with the data.

MELT introduced us to their tools, such as the Collective Conditions, an exercise called "Warming Up to Theory," in which we danced several texts that they had selected, and the "Rituals Against Barriers."

On the first day, we used "frottage" to connect with the area around us. Inspired by Shauna's guidance, we continued with "frottage" and rubbings using technologies like Lidar and photogrammetry scanning apps. This allowed us to rub objects that are usually kept behind glass, like those in display cases. We imported these rubbings into mixed reality glasses and continued rubbing with them.
On the second day, after speaking with neurosurgeon Thomas Picht and our colleagues from the Image Guidance Lab at Charité, we started thinking about brain data and our neurosurgical case study in terms of rubbing. How can we effectively rub with the spatial information to better understand it? We had a productive and relaxing time, and we're eagerly anticipating our next workshops at the Kunstgewerbemuseum. We'll be exploring how to engage visitors with the material processes and the enjoyable work of craft people through immersive installations and score. Stay tuned for more on the "Quasi-Makers" series event at KGM.

Claudia, Lin & Ren enjoying the results of the rubbing during the final public workshop session on the Friday afternoon.

Neuroscientists Melina Engelhardt and Robert Schenk grasping for brain tracts in our prototype at KGM, assisted by creative coder Warja Rybakova ©Maxime Le Calvé

Conducting a 3D sketching workshop as fieldwork: Drawing and telling neuroanatomical landmarks spatially at the Speculative Realities Lab, Charité Berlin

How does the sense of space, and the haptic creativity associated with it, affect the teaching of anatomical knowledge? How can current practices be explored by shifting them to other techniques? In this first workshop of the SpecLab, together with HCI scholar Paulina Greta Stefanovic and neurosurgeon Thomas Picht, we challenged a group of participants from neurology, neuroscience, computer science, clinical linguistics, design and art to tell us "anatomical stories" using a 3D sketching tool. After a short introduction to the research project and a short tutorial, the participants jumped into the 3D sketching action using the free program Gravity Sketch.

The technological infrastructure was difficult to deal with at first, but with some inventive ways of overcoming a problematic internet connection, we were able to have participants share virtual rooms while moving around the same physical space. The new SpecLab space is a perfect playground for collaborative 3D sketching interactions. We hope to keep it as empty and spacious as possible. With the pass-through technology of the newer standalone headsets, we didn't have to worry about collisions. This gave the typical virtual reality technology a hint of what augmented reality might look like: the user is no longer alone, secluded in the headset. A wide range of shapes and materials were tried out that day, from arterial visualisations to images of neural forests and cartoon characters.

The last part of the workshop was a "hot seat" situation without a seat: several participants volunteered to tell us the stories they had created in the 3D sketching tool. Cathy, the first participant who took the floor, started with the sketch she had made: a view of the cortex and a sketch of the hypothalamus. She explained the manipulation she is currently doing in the lab, as part of a team working on the treatment of epilepsy. I then asked her to do it again, this time sketching the story 'live' as she went through the same narrative sequence. After a little hesitation ("it wouldn't be optimal"), she responded beautifully to the challenge and sketched the elements of her demonstration again in real time. That was fun! She seemed quite happy with her performance (despite an obvious preference for a cleaner 3D image that could be imported, she said). It was an exciting first result.

Next up was visual artist Vinh, a student at KHB Weissensee. He began by changing the scale with a majestic gesture: the previous cortex sketch became a small dot in 3D space, and he immediately began drawing a folded piece of paper in the air. "This," he told us enthusiastically, "is how I get students to engage with art in museums". The students start by folding a piece of paper, then draw a part of an image they like in the exhibition and pass it on to another person who continues the drawing without knowing the previous part. Vinh demonstrated how this works by drawing in this mode for a few minutes while talking about his pedagogical work in this context.

This was a very relevant conceptual contribution to the workshop, as we were discussing different ways of making learning processes more engaging through sketching techniques. I personally felt that he was still sketching in 2D - but he started using 3D extruded shapes and then it became really interesting. What if we introduced the use of 3D sketching for museum visits, we asked ourselves in the final discussion.

This shift from flat to spatial was an important theme throughout the session: participants described tricky anatomical structures in the brain that could not be guessed from 2D images (Thomas), or drew spatial representations of neurons from historical sources to illustrate a conversation about degenerative diseases (Wendy).

Some of the participants had to be "put on the spot" to perform in front of the audience. Linda, a student on the UdK Product Design Masters programme, happened to be particularly graphically eloquent with her 3D technique. She first announced an omelette, but seemed to change her mind at the start of the narrative sequence and ended up demonstrating how to cook an egg, sunny side up, "with ketchup". Linda was obviously more familiar with 3D tools: she made an egg with a brush that creates circular shapes, and asked us how to duplicate objects (she ended up sketching every single pan). This humorous final story said a lot about the nice atmosphere that the sketching activity had created in the group: an openness that is not always easy to emulate.

In the final discussion, we reflected on the different modalities of learning through sketching - for example, sketch mapping as a way of learning a cartographic representation by drawing it many times. Sketching can be a way to disrupt the perfectionist glow of the 3D material used in scientific imagery and bring the focus back to the performance itself and the many ways it brings people together.

Special thanks to Milena Burzlaff for the amazing support!

Teaching Digital Neurosurgery as Fieldwork: Mixed Reality and 3D sketching at Speculative Realities Lab

At the Digital Neurosurgery seminar led by Prof. Thomas Picht this week and the next, we had the pleasure of hosting a demo by the software editor BrainLab. One of their researchers visited us at the Speculative Realities Lab, bringing along crates of high-tech gear. He presented to the students, and later to the staff, how the mixed reality could play in their Suite. Using Magic Leap AR glasses, a group of seven could explore a brain data set through the Elements neurosurgical planning software suite: a head of a patient with colorful tracks running through the white matter was suddenly floating in space. They multiplied, as students were invited to play around in the room with them. They could manipulate them by poking at them with a virtual laser.

This tool is introduced as a way to plan surgery, yet also as a powerful pedagogic tool for medical students. I could have a look at the questionnaires filled up the next day by the students and they highly appreciated the experience overall, most of them find it a useful addition to the program —an additional modality into the “anatomical intermediality” of medical training (Hallam 2020). The appeal of 3D images is used as a prestigious mark of futurity.

“Students ‘engage in reshaping their experiential world’, learning specialised vocabularies along with ways of seeing and acting that, it is argued, reconstitute persons – whether living patients or dead bodies – as ‘object[s] of medical attention’ ”

— Hallam, 2020, p. 102 quoting Good, 1994, 72-73

In the next room, on the other side of the wall, I was organizing a very rudimentary 3D sketching workshop using the commercially available app Gravity Sketch. The students ended up creating collaboratively a fuzzy scene, with a parrot and a palm tree, and lots of other scribbles.

This was a lot of fun, and curiosity spread as they were exchanging the headset to share what they had created with one another. The students reminisced about the white matter tracts that they had just seen on BrainLab. This same type of eye candy makes the neuroanatomy mixed reality application look so futuristic and appealing. 3D lines become a kind of play dough and the engagement was obviously higher than with the neurosurgical software. We concluded that combining both of these setups, a joyful interaction together with serious neuroanatomical content, could lead us to an even more potent pedagogical experience. During the next session, we explored the visual power in action in the digital image and how they shape medical practices (Burri 2013). We explored the multifarious ambiguities lurking underneath the apparent opposition between the “beauty” of an image and its “objectivity.” The conversation elevated as we went through a strange case: how and why is it that a medical practitioner can call a tumor “beautiful”? It feels like eye candy comes in many brands and many flavors.