A Conversation on Motion Bank

Scott deLahunta
Florian Jenett
Franz Anton Cramer

 

 

 

On 3 February 2015, Motion Bank collaborators Scott deLahunta and Florian Jenett met Franz Anton Cramer, research project “Records and Representations”, in Frankfurt/M. for a three-hour conversation on Motion Bank (www.motionbank.org). This meeting followed the online release of Motion Bank in December of 2013 and several informal exchanges between Scott deLahunta and Franz Anton Cramer on issues raised by Motion Bank and experiences made while using and discussing it in various academic and educational contexts. With regards to the focus of the research project “Records and Representations”, we particularly wanted the conversation to focus on questions of documentation, capturing, and representational / documentary status. The following text condenses a full transcript of the conversation and has been edited by all three participants.

 

 

Franz Anton: Should we consider the digital realm and the possibilities it offers as a change in the ontology of what can be done with collected data, or even facts or events? I have the impression that Motion Bank largely differs from the concept behind more traditional terminologies the dance milieu is used to and as they appear in the questions I sent in preparation of our meeting …

Scott: The relationship with the digital realm has been explored before from within the dance field, but largely towards creative and artistic ends in the form of digital tools for making and performing. Our main focus in Motion Bank was, as you included in your questions “Aufzeichnung”, the work of documenting, analysing and visualising or publishing the choreographic ideas of the guest Motion Bank artists (see http://motionbank.org/en/content/scores). We were set up to do this via direct experimentation with what computation could uniquely offer this process. So digitisation was an integral part of Motion Bank from the start. At the end of the four years, I find myself able to make the statement that in the future all forms of documentation we will be relying on for any study of movement-based forms including dance will be digitised. So we certainly could and should be looking at how that changes our understanding of things.

Franz Anton: Is there something like a “key concept” animating Motion Bank?

Scott: As I said the project had the basic aim to make all steps of this process available to computation, I would say that was its key concept. And I remember at the very beginning, Florian said, “We are thinking of everything as data.” This made an impact on me. One of the properties of this data is you get material at the digital level as components of an original source, but processed in such a way that you can re-express them in a very different form. So when you have digital video it is not just video but data. Dance is of course completely familiar with video, having been using it for years, but the idea that this is data breaks into the mainstream of dance in a new way.

Franz Anton: Florian, when Motion Bank as a project first approached you to be coordinator of the digital work on the online scores, what was the demand?

Florian: All the requirements given to us initially we ignored – we had to because they were just too broad. That’s the virtue and also the challenge of Motion Bank: it sits in the middle of big fields of interest, arts, dance, digital media, notation, science etc. So the project has an enormous surface for many people to attach their interest to, and I would have had too many constant problems to solve if we did not focus our work on the score projects with the quest artists. For example, we figured out right away when we started that it’s not possible to invent a new notation system for dance, or develop software that would be able to capture every dance. But these were the general requests or wishes. So instead of focusing on these, in line with the way we intuitively work as artists, we stepped back and decided to look at the specific content.

And ignorance was our first tool. Knowing nothing about dance, but after studying Deborah Hay’s material and hearing what Scott and Chris Roman said about it in our very first meeting we had an idea for the whole recording process. Her solos and group pieces unfold on the stage uniquely each time through the performers. They are not “repeated” at every performance. So it is not possible to record just one performance. That’s why we decided to do as many recordings as possible in order to give us something like a pool of possibilities and then look for similarities and differences between these recordings. Like footprints in the snow on a square: you can see where the trails form because people use them a lot. You would be able to compare multiple recordings and then find things that are in common and things that are very different. Once you have this pool you can think about different aspects, relate it back to the score text and use it to build an understanding of Deborah’s approach to choreography.

Franz Anton: What did the recording set-up look like?

 

image 1: Filming Setup Diagram: Florian Jenett

 

Florian: The black one is the follower then there are the four white cameras.

Scott: Someone is operating the follower, that is our close-up shot, these (indicating the two upper corners) are the two shots that are used with computer vision algorithms that determine the pathway, the position on stage (see https://vimeo.com/78797665), and this is the front static camera, the one Florian uses mostly for the annotation.

Florian: The goal was to produce what were already being called online scores. These would be there to talk about those choreographers and those choreographies and give access to them. That was the task, as we understood it. It meant for us that we needed to find different approaches for each of these scores, as they dealt with very different artists. So we did not look for a general track, not one solution that would fit all. We did try to generalise the tools we used as much as possible, but would apply them differently to the choreography.

Franz Anton: Were there any guidelines in programming the visuals, any foundations such as existing software, algorithms?

Florian: There is a lot of open source software underneath, part of which I helped develop over the years (see https://processing.org). But we did not really use anything that was already there, not as a foundation anyway. The only feature we really inherited was Piecemaker, the annotation software created by David Kern and used within the Forsythe Company (see https://vimeo.com/82728599 & http://www.piecemaker.org). We turned it upside down and inside out and then reworked it twice, so that it became part of our more general tools. Other than that, most of the stuff we did was either so basic you won’t use something existing, for example something for drawing lines. Or it was so special there was just nothing to draw upon, and we had to invent it ourselves. And that produced the two main tools we have, a new version of Piecemaker and MoSys, a content management and publishing system, actually a choreographic publishing system.

Scott: Piecemaker is the project David developed with Bill, a system to record videos in the rehearsal studio and later add annotations, type in comments and remarks. Bill used that for several pieces. It was decided we could take this as our starting point; also because there was an expectation that we would produce some general piece of software people could use to make their own scores. That’s why we decided to take what David had done as a base, refine and develop it so that it could be released more widely. The version David had programmed wasn’t in a condition to just distribute. And Florian recognised the importance of using the software to annotate Deborah’s material at a very early stage.

Florian: It was an important realisation. We knew we would need something to record abstract streams of data. We thought about having a protocol for that or using software that would do this and ended up using Piecemaker, because it has that as a foundation. We generalised it from annotations on video to basically a global timeline to which you can attach anything you have. And they will be automatically aligned in one space together. The timeline in computer terms or desktop thinking would be the project folder that contains all text files that belong to this project. One timeline per project per piece, then you attach the videos, and on the videos you attach data streams or markers for scenes or whatever is visible there – any of the metadata that you can extract from the recordings or in the recordings.

image 2: API refers to Application Programming Interface

 

 

Scott: This is a diagram you made a couple of years ago.

Florian: Time is the bold line, on top you have layers of videos, then you have markers, those can point to external data sets like 3D data, a big blob of data, or specific parts of the text or scores — that’s the basic idea of Piecemaker.

Scott: This also helps to see why synchronisation is important. We recorded so many versions of Deborah’s dance with multiple cameras, making them comparable required synchronisation.

Franz Anton: Comparable already in terms of their configuration as data, or comparable in terms of what the performers were doing?

Florian: What they were doing actually. Because the recordings of each performance of the score No Time to Fly (see http://x.motionbank.org/nttf_score/nttf_score.pdf) were different in length, varying from 30 to 50 minutes. So you cannot just put them side-by-side and have them align, that just doesn’t work. Each part of each unique performance varies in timing; so what we did is to relate the parts we recorded to specific parts in the text score so we would be able to see that what is written could be observed on the video. It allowed us to interlink specific parts of the text describing the choreography to parts of the video (see http://scores.motionbank.org/dh/#/set/april-2011-nttf-solo-recordings). We were then able to identify the same area and compare those across multiple recordings and videos. You might call them “scenes” but Deborah insists that there are no scenes, but there is a sequence of parts. We identified these parts, with her help, and these allowed us to say in part five – let’s call it Curved Path – they would all do something similar. If you compare the resulting 3D data you will actually be able to see that the movement is similar, even though different as well. To get to this comparison, you have to be able to align them, and that’s what we needed the annotation markers for.

Franz Anton: Was this way of looking at the work different from Deborah’s? Did she agree that there were similarities based on certain segments of the text? Or was it really your analytical viewpoint and Deborah might not have shared or understood it?

Florian: Both Deborah and the dancers have names for the parts of the text they use when they rehearse and discuss things. They would say ‘In Curved Path you do this and that’, so they invented a system of giving names to parts. We asked Deborah for a list of these and she gave us basically this first list of names. We used these and related them back to the text; this gave us the way of structuring the text, which actually just runs through, without any paragraphs or anything.

Scott: Deborah gave the score of No Time to Fly to each of the performers (Jeanine Durning, Ros Warby and Juliette Mapp). And they know how to work with her performance tools (see ‘four questions for the dancers’ page 3http://scores.motionbank.org/dh/#/set/introduction-to-concepts). Then they practiced on their own, at least three hours a day for three months, without communicating with each other to develop their own adaptation of the score. They came to Frankfurt in April 2011 and we recorded each of the three individual adaptations seven times. We know that in Deborah’s choreographic approach no one performance of the piece would be the definitive version — it changes every time. The one thing that’s clearly structured, however, at least from the point of view we needed, is that the score consistently goes from the beginning to the end. And, as Florian pointed out, Deborah and the performers have associating names to this sequence.

Florian: That was my way of getting into the piece. I spent almost a year setting markers at the start of each named section on those 21 recordings we made; that means about thirty for each recording. When there is not a clear end and beginning that meant trying to find the part in the performance where the transition from one part to the next was happening, and putting a marker in the middle of this area. That made me look really hard at what they were doing and after a while I would see things I couldn’t see on stage before. We then verified the markers with the dancers, and they were generally fine with my decisions. Those transitions can be hard to find though. Looking for them enabled me to understand and see a lot more of the piece than I was able to before, even though I was in the room when they performed all 21 versions. But the annotation work is very different, it is actually a way of thinking.

Scott: … thinking through annotating.

Florian: Yeah. But still we went to a show 2012 in Austin were they performed the piece, and I could not find these marker locations any more. I was trying to see when they would go from one thing into the other, but after a while I lost where they were in the score. Afterwards I approached Deborah whether she had changed the piece. She said no, it’s exactly the same.

Franz Anton: You mean it was the same score?

Florian: Yes, it was a performance of the same score, No Time to Fly, with the same structure. And I am probably the one person on this planet who looked at those 21 recordings the most times and worked with this structure really intimately. Still I can’t see it on stage. And this made Deborah happy, that I couldn’t see it.

Scott: These views suggest that even on these thin, abstract lines you see similarities and differences that are the same across all performers, all versions. From this one can speculate and question. For example, how is it that the performers, each taking the score, each knowing how Deborah works, each one making their adaptation alone with no communication, knowing that movement in time and space will vary with each performance of each adaptation — how is it that the dancers can be producing things IN time and space you can perceive of as having regularity? So for example, if you extract just the single lines of, say, four performances here of each one, you can even see they are each clearly performing the same pattern …

Franz Anton: It goes in both directions though, doesn’t it? Something you can see not in the “original” but in the Aufzeichnung, it’s very linear, very graphic, could even be on paper. Nonetheless it reveals something that maybe is not “in” the object, not as visible or not as striking – so there is a great deal of abstraction, maybe even generalisation.

 

image 3: All 21 pathways layered on top of each other,
which produces basically little useful information.

 

Scott: Time is completely collapsed. Here are all 21 versions in one view, but it’s meaningless other than they have observed one pattern here, a hot spot more or less in the centre. Further analysis needs this temporal dimension, which immediately gives you a way of thinking about the work from the perspective of ‘exploring data’. As mentioned, the goal of Motion Bank was to rigorously explore what computation and computer-based design can offer the documentation and study of dance practice. There is some fundamental confusion however about how much of this process can and should be automated. In our case, as Florian described, there was a tremendous amount of manual work involved. This is something people are not aware of, the human part. Watching for hours, using his hands and eyes to make decisions. This is a position we maintain very firmly, automation is only one tiny aspect of the process and using the computer to make things easier is a complete misconception. However, we have used the computer to help offer a completely unique view on Deborah’s work, a view that has never been seen before.

Franz Anton: If we call the score and its possible results the artefact, then still you have to be able to see them, to read them, to analyse them, to interpret them — to understand them. And then this idea of working with the data would be a means that goes beyond the basic medium, which would be the performance. It’s more than a performance …

Florian: The raw data is a separate material, its own material, which we use our tools to look into to discover things. And the artefacts actually are the things we make out of those findings again, offering different views through the online scores as a way of talking about certain elements we found. These on-line scores then become something like a tool for other choreographers to learn from, to use in education, to apply somewhere else. But the data itself, once you have it and start to work with it, I tend to refer to it as a different material.

Franz Anton: Do you consider your work documentary or analytical? Or maybe creative?

Florian: Overall the project was documentary in nature, but I specifically took an analytical view on the materials we collected, whereas Amin Weber made a performance with his own tools. He made his own digital adaptation of the No Time to Fly score as one other way of trying to get into this choreography of Deborah’s, not by recording data on stage but by trying to perform it as data. He learned the piece sort of in the way the performers learned it and would try to do one performance of this piece, and that is the 15-minute animation we have. We called this the intuitive approach. So we had the documentary approach that consisted of interviewing mainly which we did together with Scott, my analytical approach with all the data recordings and their visualisation, and Amin’s approach with the intuitive performance. That’s a threefold way into Deborah’s choreographic work…

Franz Anton: Why would Amin’s be more intuitive than your work?

Florian: Well, it is analytical too in a way but it’s just for us to say this is more intuitive when someone sets out to be creative in the first place, to develop an artistic response; my approach was to gather or collect the data first and then work with that firstly with the aim of revealing things about Deborah’s approach which could be discovered through exploring the data.

Scott: In interviews we published in Deborah’s score, Amin talks to Jeanine about how he “found a body” using his 3D tools. Florian, you worked to find ways to usefully express ideas based on the data that could correspond to Deborah’s choreographic approach. You made a lot of sketches and early visualisations.

Florian: If you look back it’s always easy to ask, “why didn’t you do this visualisation in the beginning, it shows everything?” But, it takes time to go through many versions before finding this one that works best.

Franz Anton: Coming from dance studies and dance history, I was at a given point tired of the eternal complaint that once a piece is shown it can never be brought back and you never know what it really was etcetera. So since some time I am working on the hypothesis that even if you are present, live, there in the theatre, you will never see the entire piece because it’s too much, because our sensory system is too limited, because there is just one perspective. The point I am interested in is that as you say, all of a sudden you start discovering or detecting elements or patterns or ideas, knowledge that maybe even the choreographer wasn’t aware of them.

Scott: It’s a surplus.

Franz Anton: … a surplus, exactly. And that might be an argument to fight off the discourse ‘It’s all about intuition and empathy’ and so on and so forth …

Florian: Well, that depends. For example, we said anyone who wants to work with the data, like me, would be obliged to watch one entire performance so that they understand where the data came from, its source. So building a sense of empathy and intuition from this experience. We think one should get to know the choreographer, read about them, talk to them, see them perform or someone perform their work and then decide what you are going to do with the data.

Franz Anton: Of course it’s not either-or, but the hierarchy might not be useful … the hierarchy I am referring to establishes that you have the real thing and then you have all kinds of derivatives which are more or less useful, but usually less useful. This is what I am questioning. Coming back to the markers that helped you visualise in a clearer way the various steps or segments: Who chose to visualise it by lines that resemble drawings? Maybe there is no other way of visualising?

Florian: Well, there are other ways. But it was partly a constraint.

Scott: The decision was to use regular 2D video cameras, consumer based 50 frame per second, instead of something like 3D motion capture that was used by the team at Ohio State University working with Bebe Miller and Thomas Hauert (see http://scores.motionbank.org/two/#/set/sets). From these two video streams the computer vision algorithms we talked about earlier would calculate these pathways or the lines you are talking about.

Florian: We did not have a whole 3D skeleton, but just the centre of the performers. That is in computational terms, the centre of the pixels that formed the silhouette. And this would basically just give us one dot on stage. In the first run, it was the highest pixel, and later it became the centre. So we had only one 3D point per frame, and if you connect that, it is a line.

Franz Anton: So the reason for the line is technical?

Florian: Yes, but then you could attach anything to the line. My way of working is to use the most simple and obvious way that the material gives you something as artist or designer to work with. So it was my choice to just have a line. We had people afterwards who commented ‘you could attach polygons to it’ and things like that. But I’m not really into that.

Franz Anton: There were remarks, also in a critical sense, concerning Synchronous Objects for instance, that there are all kinds of volumes and shapes, but they don’t really inform you, don’t lead anywhere really. As though it was just to show off what computation can do. So the question might not be ‘you should have done something more glamorous’; rather there was an element of choice and preference, but there was also an element of technical constraint?

Florian: It was the most obvious way of visualising it, suggested by the material itself. A sort of minimalistic approach …

Scott: The insights you get with this approach are consistent: there are three performers, and they are doing something very clear relative to their own interpretations of the score. I did not know at the beginning that we would arrive at such clear somehow simple patterns showing this. With Synchronous Objects, one of the driving questions also for Forsythe was “what else might this dance look like”. So there was a kind of interest in generating many different forms or visualisations. But none of these things claimed artistic status, and this is what makes Amin’s work with Deborah Hay’s score unique. This artistic response reflects a different kind of commitment to the development of an artwork as derived from another artwork.

Franz Anton: You said you had 21 filmings of performers doing the solo, and several cameras for each filming. What you said about the markers and the analytical work, did you have to do that for each recording from all five cameras?

Florian: Heavens no. The recordings are synchronised.

Franz Anton: So only one annotation works for all of the angles?

Florian: That’s the beauty of our rework of Piecemaker. You just add all those video streams to one timeline and find one frame as the key frame, for example ‘first step on stage’. You mark that on all five videos. Once you align those all the videos are in synch. There are some technical details, but this is the easiest way to explain how we did it. You do just one marker on the timeline, and since all the videos live on the same timeline you can basically just relate that marker to any of those recordings. Same for the soundtrack or whatever you want: it’s just one marker.

Franz Anton: And you can exchange frames from one camera to the other to edit a ‘synthetical stream’?

Florian: Sure, you can do cuts, jumping between the cameras.

Scott: That shows up on the website. You see the title here, the No Time to Fly score, and the numbers running down the right side are all the different sections, essentially these are the markers (see http://scores.motionbank.org/dh/#/set/april-2011-nttf-solo-recordings). And all the videos are aligned to the score, so if you click on the 13th section for example, all of the videos jump forward to the chosen point in time. But that marker had to be made only once for each performance.

Florian: That’s the choreographic part of the online score system we built: the tiles, although totally self-contained, are able to talk to each other, give cues, be made as counterpoints to other tiles, be made to go in synch, steer other tiles, be directed by them … Having the possibility to interrelate all the material on the website allows the editor to be sort of choreographic in the organisation of relationships.

Scott: For example roll over nr. 7 and pause, you see the name of that section “curved path” comes up. Click on that and the video will jump to that part of the score. Click on the video to start it playing, and when the performer reaches the next part of the score it jumps to the next part in the score, which is “link to portal”, all the other videos then re-align to that frame. That’s the principle …

Florian: We had the problem that we had so many recordings, three performers, 21 versions, different camera angles, nearly thirty markers – how could you help a user understand this? We came up with some complex navigation systems, but in the end I realised the simplest way would be to have a video to watch, and it would give you the time, lead you through the scenes, and all other material on that page would be linked to that one video you are watching. We removed the interface to end up with just a video, which everybody knows how to use. It drives the text, the other videos, you don’t have to talk about these relations anymore they are just there.

Franz Anton: To me this seems like a change in hierarchy. Usually you have the title of the work then you start to look at the material that goes with it. Here, in a way, it is the work itself that takes the lead, you cannot circumvent it, you have to click on the tile and it will give you the information, not someone else …

Florian: As an artist, I like this. It’s so simple.

Scott: We try to offer a deeper insight into the performers’ action, how they are making their decisions, which is interesting from the point of view of how Deborah makes work. The decisions about how they shift their attention and perception around in space – that’s what is choreographic about Deborah’s work. So we interviewed both Jeanine and Ros, as they were viewing themselves performing their adaptations, to report on what was informing their decision in those moments in performance (see http://scores.motionbank.org/dh/#/set/jeanine-ros-insights). So you can watch a full run of the performance and read the comments on the right alongside the score text running on the left.

Franz Anton: I have the impression that Motion Bank for the first time offers a ‘philological’ tool, or approach, to movement analysis. You can take a step back, it is not just the moment right now and here, not the eternal presentness. You obtain maybe a different shape, but most of all a different possibility to come back to the material. It brings dance back to a linear and scriptural context without taking away, as in Labanotation for instance, which is so far from the actual phenomenon, the simultaneity of the phenomenon, its actual realisation.

Scott: One thing we are continuing to develop is the relationship with artists whose material is data, and this means taking a decidedly artistic position on that material and its re-expression. That may appear to be more or less analytic, depending upon the perspective, but another area handling data in an analytic way for research and education purposes is the Digital Humanities. There the discussions are about being able to search from a scholarly perspective across often a big corpus of data. It’s not immediately apparent what the connections between Motion Bank and the Digital Humanities is, but people working in that domain seem interested in what we are doing. A more direct connection we make with artists working with data is through the Choreographic Coding Labs. In the CCLS, we invited artists who are skilled with code who want to work with movement. We encourage them to approach movement as a unique source of inspiration by bringing them into contact with the movement artists, for example through exposing them to the work of a choreographer like Deborah Hay. Our contention is that this movement data is richer and more interesting to work with. That’s another case where we are applying the annotation idea, because annotation not only produces data but also a semantic layer on the material. So if we give somebody Deborah’s data and encourage them to study the material we have produced through annotation on the Scores website, for example the Insights material, the idea is that they are given the chance to develop a richer understanding of the source of that movement to produce expressions of that data in an informed way. That’s a very different alignment of dance and digital media than other projects.

Franz Anton: Could one consider Motion Bank as a parameter to start Aufzeichnung not from a documentary, but an informational point of view: opening up content layers that would not exist or at least go unnoticed without the transformations and multiplications realised in the outcome; as though “the work” did not exist outside of, or independently from its computed existence?

Scott: One way of articulating the challenge is to ask how, in the absence of the presence of performance, can we communicate as much as possible about what bodies go through in the context of the experience of performing? In a way we almost reject the high-level hierarchical status: the body can only be experienced through the doing, in the presence. We are saying, ‘we don’t accept that as the fundamental relationship’. Lots of interesting things can be discovered and learned by taking a different perspective. It is important not to reduce the complexity of what happens in the studio or in the process of making, learning and performing dance. But what we are claiming is that it can be made accessible. This is not to say it is easily accessible, visiting the Scores website and the material from Deborah Hay is not an easy reading experience, so how do we encourage readers to work harder, to make use of their imagination and read between or behind the lines, literally.

Franz Anton: Probably also because it’s linear, happening in time; it seems you need to go from one chapter to the next. At the same time, you understand rather quickly that a lot of things are happening simultaneously and that even though there is linearity it is parallel. This seems to reflect very nicely the complexity of the performance, the real live performance always marked by the fact that more things are happening than can be absorbed. This element of simultaneity with comments and fragments being here and there, was it conceptual, or was it just because there was so much material that needed to be organised on the website?

Florian: We always thought of that system we built as something like a library; the part you do not see is a huge library of material, and these pages are outlets for thinking about a specific area, one view, one aspect. That’s why they are so fragmented. We have a lot of clips and versions and interview snippets and text blocks in the library. Using MoSys the publishing system, one can arrange these various assets here rather freely. That led to looking at them the way they are now. It’s more like an exhibit, with monitors, text blocks and drawings in different rooms.

Franz Anton: It strikes me that you don’t scroll down but sideways.

Florian: That’s the notion of rooms again. And this is also reflected in the material. We had a first conceptual sketch of this grid based system where one would enter the dance side from, say, the left. Then as in an exhibition, you could wander through this, go from recordings to interviews and insights to visualisations to data to software then exit on the right via the techie digital art coder side. And the other way around, the digital artists-coders could come in the right side through the data and software and end up with the performances. An exhibition with two entries: performances here on the one side, computation here on the other side and they meet in the middle. We did not make it like this in the end, but the elements are all there.

Franz Anton: The Coding Labs are usually addressed to dance related people?

Scott: They are explicitly addressed to what we call “code savvy artists” who are interested in translating dance and choreography into visual form or applying choreographic thinking to their own practice. It’s very explicit.

Franz Anton: Maybe I should ask the other way round: Who is responding? From where does the interest in this work come? Is it restrained or limited to a certain group?

Florian: It’s a mix, really. Some heard about Motion Bank and want to know more in order to do something similar. A lot of people argue they always wanted to combine software creative coding with something performative … sometimes because they have had dance and movement experience before. Today it’s pretty obvious that the connection between the human and the computer is somehow improving. Software starts to understand more abstract things; we have easier to use interfaces and a lot more possibilities with faster processors and better software. The devices are getting cheaper and cheaper, the Kinect 3D recording device is around 120 euros now. There are so many touching points. That’s why this area is so extremely important at the moment; a lot of people are looking for content. So on the creative coding side you would be seeking meaningful movement, not just something you can generate in your studio. On the other side, a choreographer would not look for just random software to generate a path from what they are doing but someone who would be able to really understand the work and do something that values that – doing software that is not just generalised for any kind of skeleton movement that would give you statistics, but something that works with your choreography, looking for understanding, looking for better tools, looking for better processes etc. There is a huge need to bridge those fields. That’s why from many sides people come to the Labs. We address the Creative Coding community mainly, so that makes up the largest portion, but we also have dance people, and we have musicians, film makers, architects – very diverse fields beginning to realise it’s a very important area to think about at the moment, and they want to spend some time with other people doing that. That’s what we try to facilitate.

Scott: So the Lab provides an environment where highly skilled people are collaborating on building an understanding of meaningful movement in relation to data. Creative work is clearly a goal, and having art making people is kind of an essential framework for the whole thing; but the aim is not necessarily to make art together. Some do, and that’s fantastic. But our goal is to develop a context were a deeper understanding of these forms – dance and data – and connections between them can be explored. One thing that has made this possible is a growing community of practice in digital media arts with the artists capable of writing code who are working with a handful of programming languages developed by artists, like processing (see https://processing.org/). That’s relatively new, since around 2000.

Franz Anton: From this point of view, digital tools for recording, storing, processing are more generative of new material; but what would be the link to documentation or archival work in the more traditional sense? Coming back to the movement aspect, you were once saying that Motion Bank is about extricating the body from movement phenomena. So the question of movement still is very complex, of course, but if it is about archiving or storing or reproducing movement that has already been done, can you do it in a medium or a context that does not move itself? We have been looking at Motion Bank under this perspective: How does it transform, translate and bring in movement? Where are the movement relationships that are direct or indirect? Is there a conceptualisation of movement that has informed your work?

Florian: I don’t have a fixed, clear concept of movement. But it is hard, almost impossible to generate this kind of movement synthetically. It is different if you have a 3D avatar animated by hand. There you produce movement, but not human movement. The aspect of not only human movement, but high quality movement in terms of the expressive aspect, makes it special and interesting to work with. If you have recorded high quality movement you can do all kinds of things on a software level and thereby generate something else. It’s just a new opportunity. And there can be a reverse effect. Take the 21 solo versions all overlaid. Deborah took that as inspiration and is now producing a version of it with 21 performers on stage. (see https://vimeo.com/77695465)

Scott: And now she has a commission from the Cullberg Ballet to make a work based on this overlays video with music by Laurie Anderson.

Florian: She said, ‘I find it hard to see my solo in some of the visualisations because they are so abstract. But I find many other things that are extremely interesting to me as a choreographer.’ The same happened with some of the tools we built for her (see https://vimeo.com/29619105). Maybe in a way the movement is broken down into particles and atoms so small it can be rebuilt to be other things. Having this sort of liquid material in your hand, that’s what she realised was interesting.

Scott: The movement dimension can be revealed through digitalisation because in both one is working with dynamic change and variables. Movement can be remediated so one can obtain different views on time and re-combinations that would be difficult, extremely difficult to do without having been digitised. I think our ability to look at choreographic work and dance, to analyse it and think about the space the work exists in, is no longer about the Cartesian coordinate space of X, Y and Z, but the space of the imagination, the space of thinking. I wonder whether these tools for probing that imagination, themselves become a way of thinking, like annotation.

Franz Anton: And it remains a body of knowledge, a corpus – even though not physical. In the 18th century, there was a debate about movement. Not artistic, but in general. Is movement a quality, or reality? Diderot posed the question, “Why shouldn’t we consider movement as a quality of matter?” It can be an aggregate state of matter; it is not necessary that matter moves but if there is movement it’s something that happens to matter. I like this Enlightenment idea about movement not just an accident or a murky phenomenon, but a quality. If this were true, the ‘traditional’ understanding of movement would be very limited because it’s considered action, maybe gesture, or a process, but always effect-oriented. But where would the quality, the essence of movement be? Maybe I am naïve, but in a certain sense I have the impression Motion Bank is a tool to consider movement as quality without always attaching it to a body, or to matter — an independence of movement, a setting free.

Scott: The MOCO (see http://moco.ircam.fr) workshops are events where mathematicians, computing scientists and engineers interested in movement qualities and in trying to model them, to describe these qualities using their own tools. When I was there a few years ago I was not only surprised at how many were quite busy with this topic, but also by how they were approaching it. Everyone was approaching the problem from different perspectives and disciplines, forming multiple and complex descriptions facilitated by computers and computation. So it makes me wonder if we are accumulating a sense of a new kind of substance around movement. And what Motion Bank offers is a view into a rich choreographic practice, as both a culture and a context for meaningful movements or non-movements as the case might be. Our claim is that the quality of this movement is extraordinary, somehow exceeding sports and everyday movement. So we try to infect thinking in this emerging movement science domain with this kind of material and with the tools for helping you probe this area that are beginning to emerge. One thing that is still very noticeable, if you read the papers that are being proposed for the next meeting of MOCO, is that the most referenced movement description systems are those attributed to Laban. What would this world of movement science look like if they were referencing Forsythe’s Improvisation Technologies or Deborah Hay’s cellular body? My position is that we need to work harder to develop the required theories and models to bring this kind of material into view as a resource for studying movement.

Franz Anton: You were saying computer is not just something that absorbs movement and then spits out something. There rather is a lot of reflection and manual labour involved. So it’s not a reduction – something I still need to learn – not a reduction of the complexity of real movement performed in reaction to a score by Deborah Hay but a transformation or translation no less complex than whatever may have been the ‘original’, just a different way of presenting it.

Florian: There is your loop back to the documentation question. We are trying to build with the tools a way to allow for complexity to be preserved. It’s the opposite of a video recording that flattens things down to an image and a soundtrack. Via meta and other forms of data, digital video material can be unwrapped and re-integrated in many different directions depending on your perspective and interests.

Franz Anton: Coming back to the intuition that Motion Bank is about simultaneity, everything being in one space, but still linear. I have the impression that compared to a traditional archival room where you have also everything ready at the same time but you cannot grasp things as easily. There might still be a parallel because you have sub-spaces in the archive – boxes, film department, photographic department etc. – and yet everything is there at the same time, physically and materially. All that is there in Motion Bank but when you consult it, you have to inscribe it on the timeline. Interestingly, the digital field doesn’t seem to work differently …

Florian: It depends what your goals are. If you are trying to just do the visualisation, you don’t care about everything in between. It is just material, with the purpose to produce an end result. But we wanted to keep everything we collected. So one reason to build this content management and publishing system – MoSys – was so Scott would be able to easily make sense out of the stuff we collected and be able to have a sort of a virtual library that he could go to.

Scott: Yes and I was essentially publishing a view on this huge corpus of material, in my selection. MoSys is unfortunately not ready for public release although this is something we are working on. But the system does respond to one of the goals of Motion Bank, which was and is to develop tools and approaches that can be applied across a range of different choreographers. As Florian mentioned earlier, there was talk about developing a new notation system, but this was never the direction of our research. Instead, Florian addressed this with certain systems, firstly further developing and using the annotation tool Piecemaker, and secondly building MoSys so that one could reach out any time into any material and flexibly produce multiple views into that material. MoSys is more like a thinking mechanism that enables the data to be accessed.

Franz Anton: Is Motion Bank less a documentation of one work but an interpretation of a score underlying the work and stipulating practice as well as thinking?

Scott: I remember that already during the first presentation of Motion Bank Florian and Amin introduced themselves as artists, not computer scientists or engineers. That radically marked the project. They established themselves as artists and said they were interested in how the dance data itself might produce new kinds of ideas about recorded dance. This was a departure from previous similar projects I worked on, but has emerged as a defining feature of the projects we will work on in the future for the second phase of Motion bank.

 

 

 

Florian Jenett is an artist and lives and works in Frankfurt, Germany. He is co-founder of the basis studio houses and art space and a longtime co ntributor to the Processing open source project. In recent years he has been researching fo r the Motion Bank, which he co-directs alongside Scott deLahunta since 2014.
http://www.florianjenett.de


Scott deLahunta has worked as writer, researcher an d organizer on a range of international projects bringing performing arts with a focus on c horeography into conjunction with other disciplines and practices. Alongside Florian Jenett , he is currently co-director of Motion Bank and Senior Research Fellow in partnership with Coventry University (UK) & Deakin University (AUS). http://www.sdela.dds.nl

 

PDF Download