The Echoes Unheard
#Interactive Storytelling #Slow Journalism #Digital Archive #Spatial Audio #Immersive Soundscape
The Echoes Unheard is an immersive web-based interactive experience that explores the poetic essence of dialects through music and personal narratives. Through captivating storytelling, it delves into the cultural richness and emotional depth embedded within various dialects.
Timeline
Tools
- February - May 2024
Tools
- Frontend Development: HTML, CSS, Three.js, Vanilla JavaScript
- Backend: Google Firebase
- Design & Modeling: Figma, Blender
My Role:
- Research & Conceptualization: Conducted extensive research to explore the intersection of dialects, music, and personal narratives
- Content Production: Produced compelling content that brings forth the poetic and cultural significance of dialects
- Interaction Design: Designed interactions to engage users and enhance storytelling immersion
- Development: Developed the frontend and backend functionalities to ensure seamless user interaction and data management
01. Inspirations & Research
My initial inpiration came from Out of Eden Walk (OOEW), a project initiated by Paul Salopek, who is a journalist and an Explorer supported by National Geographic. Paul has been walking around the world, following a historical human migration, for over 10 years, starting from South Africa and currently in China. Along the way, Paul has been writing stories about the journey and the people he has met in this journey. He has also been collecting data in various types, like images, videos, audio, panoramas, geo-locations, etc.
![]()
I had the chance to encounter with this project when I was working at Nat Geo, and my job back then was to redesign the website for OOEW, which serves as a digital archive for Paul’s writing and the data he collected.
![]()
During the project, the term “slow journalism” came up very often. It is an approach that Paul took as his practice in OOEW to record the story about people. I personally found it very touching and especially precious in this digital age where almost everything is about the speed.
So what exactly is “slow journalism”?
![]()
Erik Neveu, a journalist, defined this term from eight different aspects. Among these, four of them stood out to me the most and later became the guideline for the development of my own project:
![]()
Slow journalism has literal slowness in the processes of data collection and story invesigation.
![]()
It is about the community and for the community.
![]()
It transform its audience into partners and encourage them to respond to and contribute to the stories.
![]()
It focuses on the deep and untold backstage stories.
Out of Eden Walk, Paul Salopek
I had the chance to encounter with this project when I was working at Nat Geo, and my job back then was to redesign the website for OOEW, which serves as a digital archive for Paul’s writing and the data he collected.
Out of Eden Walk Website
During the project, the term “slow journalism” came up very often. It is an approach that Paul took as his practice in OOEW to record the story about people. I personally found it very touching and especially precious in this digital age where almost everything is about the speed.
So what exactly is “slow journalism”?
Slow Journalism, Erik Neveu
Erik Neveu, a journalist, defined this term from eight different aspects. Among these, four of them stood out to me the most and later became the guideline for the development of my own project:
Slow journalism has literal slowness in the processes of data collection and story invesigation.
It is about the community and for the community.
It transform its audience into partners and encourage them to respond to and contribute to the stories.
It focuses on the deep and untold backstage stories.
02. Conceptualization
With a deeper understanding of slow journalism in mind, I continued by asking myself:
What can I journal about?
![My Little Airport]()
![Shanghai Rainbow Chamber Singers]()
As I pondered this question, I started to realize that a lot of my favorite music were songs in dialects that I wasn’t familiar with. For instance, the left image above is one of my favorite bands, My Little Airport. They are from Hong Kong, and naturally, most of their work is in Cantonese. The one on the right is Shanghai Rainbow Chamber Singers. Similarly, a lot of their work is created and performed in Chinese dialects.
Despite the unfamiliarity from the languages, these songs are the ones that have been carrying rich personal sentiments for me.
I also discovered some common themes in them:![]()
![]()
What can I journal about?
As I pondered this question, I started to realize that a lot of my favorite music were songs in dialects that I wasn’t familiar with. For instance, the left image above is one of my favorite bands, My Little Airport. They are from Hong Kong, and naturally, most of their work is in Cantonese. The one on the right is Shanghai Rainbow Chamber Singers. Similarly, a lot of their work is created and performed in Chinese dialects.
Despite the unfamiliarity from the languages, these songs are the ones that have been carrying rich personal sentiments for me.
I also discovered some common themes in them:
Melody is indeed essential as it is the medium that allows these sentimental, poetic, and intimate sensations to flow towards me.
However, I came to realize that it was the ambiguity inbetween the familiar and unfamiliar feelings I had for these languages created all these sensations for me.
![]()
![]()
![]()
Thus, I decided that my focus would be on languages/dialects.
I also thought about my experience back in high school.
When we were learning ancient Chinese poetry, our teacher would often ask us to read those poems out in a Southern Chinese dialect, as I am personally from the southern part of China. And that is to let us feel the rhythms and rhymes in the poems, because the ancient pronunciations are better preserved in dialects than in the modern mandarin we are speaking nowadays.
I began to realize that there is something very poetic and artistic hidden within dialects.
Yet, people nowadays tend to see dialects mainly as a tool for daily communication, or even less - Due to the centralization of language and promotion of mandarin, specifically in China, a lot of dialects are being marginalized.
Therefore, my goal for this project is to reveal the
However, I came to realize that it was the ambiguity inbetween the familiar and unfamiliar feelings I had for these languages created all these sensations for me.
Thus, I decided that my focus would be on languages/dialects.
I also thought about my experience back in high school.
When we were learning ancient Chinese poetry, our teacher would often ask us to read those poems out in a Southern Chinese dialect, as I am personally from the southern part of China. And that is to let us feel the rhythms and rhymes in the poems, because the ancient pronunciations are better preserved in dialects than in the modern mandarin we are speaking nowadays.
I began to realize that there is something very poetic and artistic hidden within dialects.
Yet, people nowadays tend to see dialects mainly as a tool for daily communication, or even less - Due to the centralization of language and promotion of mandarin, specifically in China, a lot of dialects are being marginalized.
Therefore, my goal for this project is to reveal the
03. Form
Combining the findings from the research and conceptualization processes, I decided on the form that I would take to achieve my goals.
I wanted to create a collection/archive of songs that:![]()
I wanted to create a collection/archive of songs that:
In terms of its form, I decided it to be:
Combining the ideas above, this project would be:
04. Demo
a. Opening
As the user enters the website, they will see an opening scene leading them to the experience.
Headphones are recommended for spatial audio to work better. There are also instructions on navigation and interactions by keyboard and mouse.
By clicking on the boat icon![]()
, the user will begin their journey.
As the user enters the website, they will see an opening scene leading them to the experience.
Headphones are recommended for spatial audio to work better. There are also instructions on navigation and interactions by keyboard and mouse.
By clicking on the boat icon
, the user will begin their journey.b. Navigation
Once entering the scene, the user will find themselves on a boat in the water, surrounded by islands.
Each of these islands represents a song I selected that has special meanings to me. The spatial audio of that song is attached to the island.
As the user navigates and moves around, the audio changes based on their direction and distance in relation to the island.
The visual representations here like the boat, water, islands, etc., are to create a sense of drifting, being uncertain about where home is, and an always onging search for a sense of belonging.
Once entering the scene, the user will find themselves on a boat in the water, surrounded by islands.
Each of these islands represents a song I selected that has special meanings to me. The spatial audio of that song is attached to the island.
As the user navigates and moves around, the audio changes based on their direction and distance in relation to the island.
The visual representations here like the boat, water, islands, etc., are to create a sense of drifting, being uncertain about where home is, and an always onging search for a sense of belonging.
c. Journals
When the user chooses one of the islands, they will see some brief introduction to the song, including the song title, the artist, the language/dialect the song is performed in.
As they click on the book icon![]()
, a hand-written journal by me will show up. It tells a personal story between me and this song. The handwriting is in Chinese, and by hovering over the dots next to the text, tooltips for an English translation will appear.
When the user chooses one of the islands, they will see some brief introduction to the song, including the song title, the artist, the language/dialect the song is performed in.
As they click on the book icon
, a hand-written journal by me will show up. It tells a personal story between me and this song. The handwriting is in Chinese, and by hovering over the dots next to the text, tooltips for an English translation will appear.d. Audience Contribution
There is also another component where the audience are encouraged to contribute to this experience.
When clicking on the pen icon![]()
, the user will be encouraged to submit a song that has special meanings to them.
They will first see a collection of songs submitted by previous users. By hovering over the song titles, they could read about the stories related to the songs.
As they scroll down, there will be a form for them to submit their own song.
There is also another component where the audience are encouraged to contribute to this experience.
When clicking on the pen icon
, the user will be encouraged to submit a song that has special meanings to them.They will first see a collection of songs submitted by previous users. By hovering over the song titles, they could read about the stories related to the songs.
As they scroll down, there will be a form for them to submit their own song.
05. Tech Stack
For development, I used HTML, CSS, and Vanilla JavaScript for 2D interfaces, and Three.js for the 3D environment. I also used Google Firebase to store the data users submitted.
System Diagram
06. Future Opportunities
- This will be an ongoing collection of songs and stories. I hope to keep updating the collection with my personal memories and with the contribution from my audience as well.
- I hope to experiment with visualizing the data from audience contribution by auto-generating an island when there is a new submission.
- I am also curious about the opportunity to integrate WebXR, combing spatial audio with a headset to create a more immersive soundscape experience.
