The Echoes Unheard

#Interactive Storytelling    #Slow Journalism    #Digital Archive    #Spatial Audio    #Immersive Soundscape

The Echoes Unheard is a web-based interactive storytelling experience that unveils the poetic essence of dialects through the lens of music and personal stories.
Timeline: February - May 2024

Tools: HTML, CSS, Three.js, Vanilla JavaScript, Google Firebase, Figma, Blender
My Role:
  • Research & Conceptualization
  • Content Production
  • Interaction Design
  • Development

01. Inspirations & Research

My initial inpiration came from Out of Eden Walk (OOEW), a project initiated by Paul Salopek, who is a journalist and an Explorer supported by National Geographic. Paul has been walking around the world, following a historical human migration, for over 10 years, starting from South Africa and currently in China. Along the way, Paul has been writing stories about the journey and the people he has met in this journey. He has also been collecting data in various types, like images, videos, audio, panoramas, geo-locations, etc. 

Out of Eden Walk, Paul Salopek

I had the chance to encounter with this project when I was working at Nat Geo, and my job back then was to redesign the website for OOEW, which serves as a digital archive for Paul’s writing and the data he collected.

Out of Eden Walk Website

During the project, the term slow journalism came up very often. It is an approach that Paul took as his practice in OOEW to record the story about people. I personally found it very touching and especially precious in this digital age where almost everything is about the speed.

So what exactly isslow journalism”?

Slow Journalism, Erik Neveu

Erik Neveu, a journalist, defined this term from eight different aspects. Among these, four of them stood out to me the most and later became the guideline for the development of my own project:

Slow journalism has literal slowness in the processes of data collection and story invesigation.

It is about the community and for the community.

It transform its audience into partners and encourage them to respond to and contribute to the stories.

It focuses on the deep and untold backstage stories.

02. Conceptualization

With a deeper understanding of slow journalism in mind, I continued by asking myself:

What can I journal about?

As I pondered this question, I started to realize that a lot of my favorite music were songs in dialects that I wasn’t familiar with. For instance, the left image above is one of my favorite bands, My Little Airport. They are from Hong Kong, and naturally, most of their work is in Cantonese. The one on the right is Shanghai Rainbow Chamber Singers. Similarly, a lot of their work is created and performed in Chinese dialects.

Despite the unfamiliarity from the languages, these songs are the ones that have been carrying rich personal sentiments for me.
I also discovered some common themes in them:

Melody is indeed essential as it is the medium that allows these sentimental, poetic, and intimate sensations to flow towards me. 
However, I came to realize that it was the ambiguity inbetween the familiar and unfamiliar feelings I had for these languages created all these sensations for me.

Thus, I decided that my focus would be on languages/dialects

I also thought about my experience back in high school.
When we were learning ancient Chinese poetry, our teacher would often ask us to read those poems out in a Southern Chinese dialect, as I am personally from the southern part of China. And that is to let us feel the rhythms and rhymes in the poems, because the ancient pronunciations are better preserved in dialects than in the modern mandarin we are speaking nowadays.

I began to realize that there is something very poetic and artistic hidden within dialects. 

Yet, people nowadays tend to see dialects mainly as a tool for daily communication, or even less - Due to the centralization of language and promotion of mandarin, specifically in China, a lot of dialects are being marginalized. 

Therefore, my goal for this project is to reveal the 

03. Form

Combining the findings from the research and conceptualization processes, I decided on the form that I would take to achieve my goals.

I wanted to create a collection/archive of songs that: 
In terms of its form, I decided it to be:

Combining the ideas above, this project would be:

04. Demo

a. Opening
As the user enters the website, they will see an opening scene leading them to the experience. 

Headphones are recommended for spatial audio to work better. There are also instructions on navigation and interactions by keyboard and mouse.

By clicking on the boat icon , the user will begin their journey.

b. Navigation

Once entering the scene, the user will find themselves on a boat in the water, surrounded by islands.

Each of these islands represents a song I selected that has special meanings to me. The spatial audio of that song is attached to the island.
As the user navigates and moves around, the audio changes based on their direction and distance in relation to the island.

The visual representations here like the boat, water, islands, etc., are to create a sense of drifting, being uncertain about where home is, and an always onging search for a sense of belonging.

sound on

c. Journals

When the user chooses one of the islands, they will see some brief introduction to the song, including the song title, the artist, the language/dialect the song is performed in.
As they click on the book icon , a hand-written journal by me will show up. It tells a personal story between me and this song. The handwriting is in Chinese, and by hovering over the dots next to the text, tooltips for an English translation will appear.

sound on

d. Audience Contribution

There is also another component where the audience are encouraged to contribute to this experience.
When clicking on the pen icon , the user will be encouraged to submit a song that has special meanings to them.

They will first see a collection of songs submitted by previous users. By hovering over the song titles, they could read about the stories related to the songs.

As they scroll down, there will be a form for them to submit their own song.

05. Tech Stack

For development, I used HTML, CSS, and Vanilla JavaScript for 2D interfaces, and Three.js for the 3D environment. I also used Google Firebase to store the data users submitted.

System Diagram

06. Future Opportunities 

  • This will be an ongoing collection of songs and stories. I hope to keep updating the collection with my personal memories and with the contribution from my audience as well.

  • I hope to experiment with visualizing the data from audience contribution by auto-generating an island when there is a new submission.

  • I am also curious about the opportunity to integrate WebXR, combing spatial audio with a headset to create a more immersive soundscape experience.