CURIOSITY & PLEASURE : FEELING & MEASURING

A Data-Driven Composition Based on Affective Ratings During Music Listening (Arvo Pärt’s Fratres)

Collaborators:

• • Antoni Rodriguez-Fornells (U Barcelona, ICREA, IDIBELL, IMERA), Gemma Cardona (U Barcelona), Xim Cerdà (U Barcelona), Olivier Penacchio (U Barcelona), Laura Ferreri (U Pavia; U Lumière Lyon 2), Robert J. Zatorre (McGill).

Versions of Pärt’s Fratres —the composition this project explores— are heard underscoring charged moments in multiple film soundtracks…

 

[Audio examples will be added soon.] 

This project was initiated during a transdisciplinary fellowship with the IMERA Institute of Advanced Study and the Institute of Cancer and Immunology, as part of a broader exploration into multimodal data mapping in health and science contexts.

The mapping here centres on the sonification of real-time affective data collected in a study on musical curiosity and pleasure. The resulting composition transforms perception data into sound—raising questions about recursive listening, compositional agency, and audience response.

The study itself, designed and conducted by Antoni Rodriguez-Fornells and collaborators, asked participants to continuously rate either their level of curiosity or pleasure while listening to two live performances of Arvo Pärt’s Fratres. The aim was to investigate potential temporal—and possibly causal—relationships between these two affective states.

For me, the resulting dataset—second-by-second self-ratings—offered a rich opportunity to explore how subjective responses to music might themselves be translated into musical form: a kind of meta-composition, or data-driven music about music perception.

1. Sonifying the Data

Dataset Characteristics

• Participants gave continuous self-ratings during live performances of Fratres
• Two affective conditions: curiosity and pleasure
• Data recorded at 1Hz resolution (one rating per second)
• Over 200 participants per condition

Preprocessing: 

The dataset was generated and analyzed independently by the research team —I stepped in after those stages of data collection and analysis.

I explored how the data could be translated into musical form. I was curious to create a sonic expression that both conveys the data coherently and resonates with the music to which the data refers—telling a second-order perceptual story.

Here’s how I’ve approached it…

Vertical Downsampling

• For each time point, the 200+ individual ratings per affect were averaged
• This created two continuous time series (one for each affect)
• These were treated as two separate melodic contours in the sonification

Pitch Mapping

• Changes in rating values were mapped to musical intervals:
  • Rising rating → rising pitch
  • Falling rating → falling pitch
  • No significant change → sustained pitch
• Normalized values were scaled onto a custom pitch scale designed to harmonize with Fratres

Layering with the Original Composition

• Sonification aligned to the original Fratres recording for solo violin, string orchestra, and percussion
• Contrasting orchestration strategies used to separate affective lines:
  • e.g., brass vs woodwinds, or piano vs bell tones (to contrast original strings and percussion)
• Stereo and binaural spatialization enhanced perceptual separation

2. Transition to Standalone Version

Soon I became curious: could the sonification stand on its own, without the original piece? As Fratres’ Frater, Soror, or Geminus? Would the structure of collective psychological response made audible function as a compelling composition in its own right?

Standalone Structure

• Curiosity mapped to low strings, panned left
  
• Pleasure mapped to high strings, panned right
  
• Removed Pärt’s original audio entirely
  

Challenges

• Without Pärt’s harmonic and dynamic narrative, the sonified data initially felt too static
  
• Participants often maintained the same rating across multiple seconds; after aggregation and pitch mapping, this led to long stretches of repeated notes
  
• The melody was noticeably constrained to a fixed 1Hz pulse, further limiting rhythmic variation

Solutions

• Experimented with interpreting repeated values as tied notes, introducing phrasing and contour
  
• Introduced tempo variation (time-stretching and compressing sections) to inject movement and emphasize internal drama
• Result: a shorter, more dynamically shaped standalone piece

3. Variations and Extensions

Polyphonic Individual Traces

• Instead of averaging, used individual participants’ data as separate melodic lines
  
• Resulted in:
  
  • Denser textures
    
  • More faithful representation of inter-subjective variability
  • But also increased complexity that reduced clarity

Potential Third-Order Layer

• Considered adding a new musical layer:
  
  • Improvised or composed, responding to the sonified data
    
  • Could serve as a performer’s interpretive commentary on perception—and on perception as data

4. Reflections and Open Questions

What emerges is a kind of meta-music: a second-order composition derived from the real-time psychological responses of listeners to another musical work. It’s compelling to hear how these traces—collective fluctuations in attention, curiosity, and pleasure—can themselves take on musical form.

Antoni asked whether we could test if derivative pieces would evoke comparable or predictable trajectories in new listeners through the very structure of the data it sonifies? And consequently, might affective relationships be explored in third-, fourth-, or even x-order compositions?

I wonder if such processes could point toward new possibilities in therapeutic composition—music designed with intentional psycho-emotional effects? These remain open questions.

5. Broader Context and Future Applications

The data gleaned in this study could also be compared with results from similar contexts to examine how different controls and variables affect engagement ratings.

• Fratres exists in multiple arrangements (solo, ensemble, duo), each with differences in:
  • Register
  • Instrumental texture
  • Expressive dynamics
    
• How might these differences affect listener engagement and affective response?
  
• Fratres is also used (some say overused) to score dramatic moments in multiple films and documentaries…
  • Could audiovisual context modulate the same emotional contours?
    
  • This could be explored artistically, though it may exceed what traditional scientific protocols can control for

In the clips above one sees how Fratres is used powerfully in films such as There Will Be Blood and The Place Beyond the Pines, underscoring moments of profound moral conflict and reckoning with greed and corruption. These instances further suggest the composition’s capacity to evoke complex affective landscapes—whether in narrative cinema or empirical research contexts (a bit more on this below).

6. Feeling & Measuring

The objective of the original study—and the resulting data—is to better understand the mind, and how we engage with music, in terms of curiosity and pleasure. My experiments ask whether and how this data might be translated coherently into a sound which echoes the musical feeling and structure of the original.

A sensitive performer is often deeply attuned to contextual cues—think of Keith Jarrett, or another powerful live artist’s relationship with their audience. A framework for improvisation built around audience reaction mediated through data may, to some performers, seem questionable—or even absurd:

I read the room!
…Why not just watch and listen to the audience to know how they feel in real time? I don’t need you to ask them—they tell me with their faces, posture, breath, and all the sounds they make …or don’t make.

And yet, from scientific and philosophical standpoints, there are artistically compelling aspects to using data to construct recursive affective dialogues between perception and sound. This recursive logic also sharpens the relevance of a third-order musical layer—improvised or composed—as a performative response to prior listeners’ emotional contours.

As an artist, I reflect on the study’s original question through lived experience. I’ve often wondered whether my discipline and commitment to exploration prioritize curiosity over immediate pleasure—seeking and delivering moments of discovery and insight that others might not reach without guidance.

Is this the double-edged sword of professional rigor? It can risk severing ties to ‘beginner’s mind’, where unfiltered sensory pleasure often resides. Re/integrating curiosity and pleasure may be a powerful key towards fulfillment—but the idealized balance may also be a matter of idiom, taste, and context.

To this point, it’s worth noting that Fratres—meaning fraternity or brethren—itself unfolds through an algorithmic, mathematically structured process: Pärt’s tintinnabuli  compositional technique, designed to evoke a sacred, spiritual, and emotionally resonant experience. (In fact, tintinnabuli actually dictates certain musical voices are restricted to certain group of notes, a parallel to the sonification scheme used here mapping curiosity or pleasure to distinct instrumental voices… though I could go further and map the data ratings to the identical groups of notes as restricted in each voice of Pärt’s original composition.)

So even before this project of collecting data about the psycho-emotional experience of the music, and even before sonifying that data in a recursive echo of the original, Fratres is already a mathematically precise exploration of affect: spiritual, emotional, and perceptual, rendered through sound.

That it is so often used in film to underscore scenes of moral struggle, spiritual reckoning, and fractured fraternity—as in the muddy confrontation between Daniel and Eli in There Will Be Blood, or the quiet interrogation of Avery, a wounded cop, in The Place Beyond the Pines—only deepens its role as a vessel for navigating corruption and conflict.

These cinematic moments don’t just showcase the expressive power of Fratres—they mirror the very questions that empirical research tries to answer: how music moves us, how structure carries emotion, and how sound becomes meaning.

[Audio examples will be added soon.]