Rubbing as fieldwork: Workshop on site writing with technologies of captures, with MELT, Shauna Janssen and the Speculative Realities Lab at Kunstgewerbemuseum

Last week, in collaboration with the SpecLab, I had the privilege of inviting MELT (Ren Loren Britton + Iz Paehr) and Shauna Janssen from Concordia University for a three-day workshop/gathering at the Kunstgewerbemuseum, with Claudia Banz as our host. The vibrant group of participants and the museum's cosy atmosphere created a splendid contrast against its spacious and brutalist architecture.
Ren and Iz led the morning sessions, while Shauna unfortunately had to remain at home. However, her insights on “Site Writing” continued to resonate with us throughout the workshop. In the afternoons, the SpecLab team presented multiple resources related to our ongoing research on brain data navigation at Charité. The focal point of this presentation was the early-stage prototype itself, which can be seen in the image above, as neuroscientist Melina Engelhardt interacts with the data.

MELT introduced us to their tools, such as the Collective Conditions, an exercise called "Warming Up to Theory," in which we danced several texts that they had selected, and the "Rituals Against Barriers."

On the first day, we used "frottage" to connect with the area around us. Inspired by Shauna's guidance, we continued with "frottage" and rubbings using technologies like Lidar and photogrammetry scanning apps. This allowed us to rub objects that are usually kept behind glass, like those in display cases. We imported these rubbings into mixed reality glasses and continued rubbing with them.
On the second day, after speaking with neurosurgeon Thomas Picht and our colleagues from the Image Guidance Lab at Charité, we started thinking about brain data and our neurosurgical case study in terms of rubbing. How can we effectively rub with the spatial information to better understand it? We had a productive and relaxing time, and we're eagerly anticipating our next workshops at the Kunstgewerbemuseum. We'll be exploring how to engage visitors with the material processes and the enjoyable work of craft people through immersive installations and score. Stay tuned for more on the "Quasi-Makers" series event at KGM.

Claudia, Lin & Ren enjoying the results of the rubbing during the final public workshop session on the Friday afternoon.

Neuroscientists Melina Engelhardt and Robert Schenk grasping for brain tracts in our prototype at KGM, assisted by creative coder Warja Rybakova ©Maxime Le Calvé

Conducting a 3D sketching workshop as fieldwork: Drawing and telling neuroanatomical landmarks spatially at the Speculative Realities Lab, Charité Berlin

How does the sense of space, and the haptic creativity associated with it, affect the teaching of anatomical knowledge? How can current practices be explored by shifting them to other techniques? In this first workshop of the SpecLab, together with HCI scholar Paulina Greta Stefanovic and neurosurgeon Thomas Picht, we challenged a group of participants from neurology, neuroscience, computer science, clinical linguistics, design and art to tell us "anatomical stories" using a 3D sketching tool. After a short introduction to the research project and a short tutorial, the participants jumped into the 3D sketching action using the free program Gravity Sketch.

The technological infrastructure was difficult to deal with at first, but with some inventive ways of overcoming a problematic internet connection, we were able to have participants share virtual rooms while moving around the same physical space. The new SpecLab space is a perfect playground for collaborative 3D sketching interactions. We hope to keep it as empty and spacious as possible. With the pass-through technology of the newer standalone headsets, we didn't have to worry about collisions. This gave the typical virtual reality technology a hint of what augmented reality might look like: the user is no longer alone, secluded in the headset. A wide range of shapes and materials were tried out that day, from arterial visualisations to images of neural forests and cartoon characters.

The last part of the workshop was a "hot seat" situation without a seat: several participants volunteered to tell us the stories they had created in the 3D sketching tool. Cathy, the first participant who took the floor, started with the sketch she had made: a view of the cortex and a sketch of the hypothalamus. She explained the manipulation she is currently doing in the lab, as part of a team working on the treatment of epilepsy. I then asked her to do it again, this time sketching the story 'live' as she went through the same narrative sequence. After a little hesitation ("it wouldn't be optimal"), she responded beautifully to the challenge and sketched the elements of her demonstration again in real time. That was fun! She seemed quite happy with her performance (despite an obvious preference for a cleaner 3D image that could be imported, she said). It was an exciting first result.

Next up was visual artist Vinh, a student at KHB Weissensee. He began by changing the scale with a majestic gesture: the previous cortex sketch became a small dot in 3D space, and he immediately began drawing a folded piece of paper in the air. "This," he told us enthusiastically, "is how I get students to engage with art in museums". The students start by folding a piece of paper, then draw a part of an image they like in the exhibition and pass it on to another person who continues the drawing without knowing the previous part. Vinh demonstrated how this works by drawing in this mode for a few minutes while talking about his pedagogical work in this context.

This was a very relevant conceptual contribution to the workshop, as we were discussing different ways of making learning processes more engaging through sketching techniques. I personally felt that he was still sketching in 2D - but he started using 3D extruded shapes and then it became really interesting. What if we introduced the use of 3D sketching for museum visits, we asked ourselves in the final discussion.

This shift from flat to spatial was an important theme throughout the session: participants described tricky anatomical structures in the brain that could not be guessed from 2D images (Thomas), or drew spatial representations of neurons from historical sources to illustrate a conversation about degenerative diseases (Wendy).

Some of the participants had to be "put on the spot" to perform in front of the audience. Linda, a student on the UdK Product Design Masters programme, happened to be particularly graphically eloquent with her 3D technique. She first announced an omelette, but seemed to change her mind at the start of the narrative sequence and ended up demonstrating how to cook an egg, sunny side up, "with ketchup". Linda was obviously more familiar with 3D tools: she made an egg with a brush that creates circular shapes, and asked us how to duplicate objects (she ended up sketching every single pan). This humorous final story said a lot about the nice atmosphere that the sketching activity had created in the group: an openness that is not always easy to emulate.

In the final discussion, we reflected on the different modalities of learning through sketching - for example, sketch mapping as a way of learning a cartographic representation by drawing it many times. Sketching can be a way to disrupt the perfectionist glow of the 3D material used in scientific imagery and bring the focus back to the performance itself and the many ways it brings people together.

Special thanks to Milena Burzlaff for the amazing support!

Teaching Digital Neurosurgery as Fieldwork: Mixed Reality and 3D sketching at Speculative Realities Lab

At the Digital Neurosurgery seminar led by Prof. Thomas Picht this week and the next, we had the pleasure of hosting a demo by the software editor BrainLab. One of their researchers visited us at the Speculative Realities Lab, bringing along crates of high-tech gear. He presented to the students, and later to the staff, how the mixed reality could play in their Suite. Using Magic Leap AR glasses, a group of seven could explore a brain data set through the Elements neurosurgical planning software suite: a head of a patient with colorful tracks running through the white matter was suddenly floating in space. They multiplied, as students were invited to play around in the room with them. They could manipulate them by poking at them with a virtual laser.

This tool is introduced as a way to plan surgery, yet also as a powerful pedagogic tool for medical students. I could have a look at the questionnaires filled up the next day by the students and they highly appreciated the experience overall, most of them find it a useful addition to the program —an additional modality into the “anatomical intermediality” of medical training (Hallam 2020). The appeal of 3D images is used as a prestigious mark of futurity.

“Students ‘engage in reshaping their experiential world’, learning specialised vocabularies along with ways of seeing and acting that, it is argued, reconstitute persons – whether living patients or dead bodies – as ‘object[s] of medical attention’ ”

— Hallam, 2020, p. 102 quoting Good, 1994, 72-73

In the next room, on the other side of the wall, I was organizing a very rudimentary 3D sketching workshop using the commercially available app Gravity Sketch. The students ended up creating collaboratively a fuzzy scene, with a parrot and a palm tree, and lots of other scribbles.

This was a lot of fun, and curiosity spread as they were exchanging the headset to share what they had created with one another. The students reminisced about the white matter tracts that they had just seen on BrainLab. This same type of eye candy makes the neuroanatomy mixed reality application look so futuristic and appealing. 3D lines become a kind of play dough and the engagement was obviously higher than with the neurosurgical software. We concluded that combining both of these setups, a joyful interaction together with serious neuroanatomical content, could lead us to an even more potent pedagogical experience. During the next session, we explored the visual power in action in the digital image and how they shape medical practices (Burri 2013). We explored the multifarious ambiguities lurking underneath the apparent opposition between the “beauty” of an image and its “objectivity.” The conversation elevated as we went through a strange case: how and why is it that a medical practitioner can call a tumor “beautiful”? It feels like eye candy comes in many brands and many flavors.

Awake Surgery as Fieldwork: snippet from the "Sketching Brains" participant exhibition project at Charité Berlin

Awake surgery has been a hot spot of neurosurgical research and a growing site of interest for journalists and scholars. The idea is simple yet radical. The patient has a tumor that lies close to the “eloquent areas.” Removing this tumor puts at risk their cognitive abilities, in particular those related to speech. Instead of waiting for them to wake up in order to assess potential damage, which may then be irreversible, why not keep them wide awake during the process, and simulate with low-current electricity the effect of potential lesions on the brain? The result is a very atypical surgical situation with a speaking patient, a patient which is aware of what is happening in the room. This calls for a mastery of the atmosphere around the patient, an atmosphere that requires perfectly rounded-up teamwork to even out possible stressful situations. Here is a draft of one of the graphic ethnographic “series,” that I am presenting in the “Sketching Brains” participant exhibition at Charité in Berlin.

Credit MLC

That morning as I follow Katharina Faust to the OR, she tells me about the “very big tumour” she will have to expurgate from the patient’s brain. The patient is an artist. The OR is quiet, that particular patient seems quite sedated. Katharina bends over the patient, and with a very soft voice, greets her and asks her how she’s feeling today. Mehmet Tuncer, a junior resident who will be stay in contact with the patient during the whole operation, is already at her side. He has already built a relation of trust with the lady? The sounds are ushed and the movements are quiet.

Credit MLC

Three students are in the room this morning: the OR keeps it character of theater and it is very usual to have a small audience for the neurosurgeons at the University Hospital of Charité, which is the leading training center in Germany and in Europe. A few more signatures for Katharina, who seems to be signing papers all day, even a few away from a feat of surgical mastery.

Credit MLC

The skull of the patient is held into a metal clasp, between two dented spikes, as in any other neurosurgical intervention. What is usually a routine affair of a few seconds take longer when the patient is awake, as this is one of the least comfortable feature of the intervention: feeling one’s head stuck into a jaw of steel. A decent amount of local anesthetics makes it a bit easier. The instrument is shown to the patient to reduce her anxiety, with a few sympathetic words.

Credit MLC

A little electric trimmer buzzes in the hands of Katharina as she clears the space where she will incise the scalp. At the site where the outside world is about to smuggle in, a generous spraying of antiseptic solution is keeping the danger of a bacterial incursion at bay. 

The head of the patient is also “registered” for the virtual representations of the screen to match with the actual body. A stereoscopic set of cameras is helping the system keep track of the scene, orienting itself of the bearing of the four balls assembled into a cross. On the three big screens of the OR, the scans show the bulky tumour in dark, dangerously close to the colorful lines hinting at the presence of language tracks pathways in the connective tissues of the white matter. 

Credit MLC

When the trimming is done, Dr Faust covers the patient’s head with adhesive blue paper protections. Only the site of the craniotomy will be apparent from this side. On the opposite side of the bed, an assemblage of machines has taken over part of the vital functions of the body for the length of the operation.

Credit MLC

My tablet is running out of battery. The last image I can sketch is in the middle of the langage mapping: as Katharina is touching the brains with a pair of electrodes, triggering a temporary inactivation at a determined point, Mehmet is showing to the lady a set of images. Every time Mehmet says “anomie,” it means that the word, like “hand” or “cat,” came out wrong from the patient’s mouth. Katharina then duly puts a numbered piece of paper on the area she just probed, and registers it into the machine with a long pointer. Another round of images focuses on actions. This series of linguistic tasks, coming straight from neuroscientific experimental processes, makes it possible to assess if that specific part of the cortex is critical to the speech functions. The lady is getting tired, but the key information is in. The resection can start! It’s a impressive moment which I won’t be able to render today in the graphic form, as my iPad calls it a day. Next operation is two days, later. That’s exciting!

"Para-siting" Fieldwork: video interview sessions with brain scientists at Charité for a collaborative project at the Humboldt Lab

This week we started the shooting of a movie that will present the activities of the Image Guidance Lab at the Humboldt Forum in Berlin. Thomas Picht asked for my contribution as an ethnographer to the project. Recognizing an apt opportunity for intervention and invention with these research partners, I negotiated (not long, fortunately!) to combine this short project with a longer ethnographic video project. George Marcus labeled this kind of opportunity as a “para-site” (2000): a space aside from the everyday life of the community that I am observing, in which we are given the chance to think and perform together, and by doing this, to build a shared understanding the reality of the collective research at the lab and present it to the world.

Doctoral researchers Melina Engelhardt and Mehmet Tuncer engaged with me to develop this project, together with Yoonha Kim. First, in the row, Mehmet shared his perspective on his current research and on the future of neurosurgical planning. We sat in front of the green screen that is now used for the current COVID online teaching efforts, under the supervision of our cameraman Theo. Melina took a photograph to share both on her Twitter account and the labs. That picture helped me to compose this scene from another perspective.

Thanks to the cluster Matters of Activity, we could hire an experienced and talented filmmaker, Theo Thiesmeier, who has a long-standing practice of producing movies in curatorial contexts. We sought and found, thanks to Thomas, a beautiful space to host a series of group interviews: the historical library of the not-less-historical Rudolf Ludwig Carl Virchow anatomical auditorium. On the walls of the library, a collection of skulls and mortuary masks of celebrities of that time. Two skeletons were hanging out there with us. But they don’t get to feature in our movie… “a bit too weird”, says Theo. Maren Denker accepted generously to give us a short interview there about her nascent project at the Lab on the improvement of surgical treatment of facial paralysis.

The “main” of this first film project around the Digital Twin project is an interview with its group leader, Lucius Fekonja. I have already written a first article on his brain cartographic practice, focusing on his intersecting expertise as a medical illustrator and neuroscientist (soon accessible online there). However, the camera and this specific project gave us the unforeseen possibility of sitting and conversing together for more than 90 minutes. We discussed the installation in the Humboldt Lab, his current project around the improvement of planning surgical intervention on speech functions. He was also kind enough to unfold his medical illustration working process for us, working through the layers on his giant drawing tablet… more on this very soon! “What I like about Maxime’s project is that it is somewhere midway between an art and science format”, he said to Theo. Of course, he appreciates the interest that I manifest for his work. Like him, I see this collaboration as the exchange of time and ultimately as the “exchange of gifts”, as the ethnographer and film maker John L. Jackson puts it (2004). An exchange of gifts that is intertwining our careers and our destinies.

This fusion between the pictures of Lucius and the works of Mehmet hasn’t happened yet in the video, but it is soon to come. This is also what multimodal anthropology is leading to —an inventive practice which is first and foremost a collaborative thing going on between the research partners and the anthropologist. This process gives shape to surprising results and engaging results between ethnography and fiction, between the mediation of a present reality, a reflection on the shape of a practice, and a shared speculation on the future.

Fieldwork at the Image Guidance Lab (Charité): how to "preplan" a neurosurgical operation

Today, Thomas Picht and Lucius Fekonja were teaching a group of seven students how to prepare a brain surgery. All of it happens on Brainlab, a software that is considered the “Apple” among the medical software, says Thomas.

The software is available from any computer in the hospital. The data of the MRI appears directly in the program. First part is to find the patient and access to the relevant images.

The next step is a fusion of the images that are available.

The scan appears in three dimensions, according to the series of images generated by the MRI. From those 3 series, the software generates a 3D visualisation, on the up-left corner.

The next step is to “draw” the tumor of the patient. Several tools exists in the program. On one of the three 2D images, the student “draws” the tumor by manually highlighting the borders of that slightly zone. An algorithm in the software guides this drawing by coloring automatically the tumor. The “drawing” appears as one “slice” of the tumor in the 3D representation.

After having done the same work on another 2D, the tumor appears now in 3D. the student drawing on a second of the 2D images, the tumor now appears as a 3D representation in the viewer. This has to be refined now, and depending on the clarity of the definition of the tumor, the manual result can show a sensible difference with the result of the algorithm. Thomas explains that even very experienced users can obtain slightly different drawings: the challenge is that the border is never completely sure. They discuss now about how to access that tumor, there are several possibilities with various difficulties.

What can we do after this, asks Thomas? “Tractography!” reply the students. They have been learning the day before from Lucius about that relatively new technique of visualization of the most important networks of the white matter: those related to motric and speech functions. Those “circuits” are statistically computed from MRI data (that’s for another post). Most importantly, drawing those tracks enable the surgeon to avoid damaging them during the operation.

This track, in blue, was highlighted successfully by the student. “When the track is so neatly appearing on the visualization software, you can be sure that it’s an important one”, says Thomas.

That was it for the day. Last discussion, Thomas explains about their current research interest in more advanced modes of data visualisation. Was the software clear enough? Do we need an hologram to visualise the objects in 3D? The opinions are divergent. But all seem very satisfied by this thourough hands-on introduction. Thomas conclude: “The surgeons do exactly the same that you did today. They only do it a bit faster!”