Bon appétit

Beyond the Bloom

An interactive folktale exhibition where visitors become the protagonists.

Floruga|Yudam ShinYudam ShinGeon KimGeon KimYoungjun ParkYoungjun ParkJisoo LeeJisoo LeeEunhyun SeoEunhyun Seo

병풍연화가로포스터.png

Beyond the Bloom is an interactive XR storytelling exhibition where visitors’ gestures become stories, and those stories are transformed into panels of a digital folding screen. The project draws on two familiar sources: traditional folktales and the childhood experience of creating story-based folding screens. While folktales are widely known, they are often viewed as children's literature, and folding screens are considered historical artifacts disconnected from contemporary cultural experiences. Beyond the Bloom reconnects these elements through generative AI and body-based interaction, reimagining folktales not as stories to be observed, but as stories to be entered, altered, and co-created. Upon entering the exhibition, visitors are introduced to the narrative and have their faces photographed before becoming participants in the story itself. They interact with a media wall through bodily movement, which is interpreted as actions within the narrative. These actions influence the course of the story, leading to alternative scenes and endings that diverge from the original tale of Simcheongjeon. At the conclusion of the experience, a new folktale scene incorporating the visitor’s appearance and actions is generated as a folding-screen image and displayed within the exhibition space. Each participant contributes a single narrative fragment, and as more visitors take part, these fragments accumulate into a collectively created folktale. The work seeks to create a space where the emotional resonance of traditional folklore coexists with contemporary digital technologies. Visitors become more than passive viewers; they become co-authors who shape the narrative through their movements and choices. Through this process, Beyond the Bloom poses a simple yet enduring question: How can stories from the past become stories of our own?

Technical Description

Project Technologies and System Architecture Beyond the Bloom is an interactive XR exhibition based on Korean folktales. It is designed as a participatory storytelling system in which visitors’ physical actions directly influence the progression and outcome of the narrative. Visitors assume the role of a character within the story, experiencing a version of the narrative that evolves according to their movements and choices. Upon completing the experience, a folding-screen artwork incorporating both the visitor’s facial features and narrative outcome is generated and archived within the exhibition space as well as on an online platform. The project consists of three primary components: a real-time interaction system, an AI-driven content production pipeline, and an AI-powered folding-screen image generation system. The real-time interaction system is built using Unity and Azure Kinect. Azure Kinect continuously tracks the visitor’s three-dimensional skeletal data, while Unity retargets this data onto characters within the folktale environment. This process transforms visitors from passive spectators into active participants, allowing them to directly shape the unfolding narrative. Story branching is enabled through a custom-developed Action Recognition system. The model analyzes skeletal data in real time to identify specific gestures and actions, which are then used as branching conditions within the narrative. While all visitors begin with the same folktale, their actions lead to different scenes and endings, resulting in a personalized storytelling experience. The characters and cinematic sequences featured in the exhibition were produced through a generative AI workflow. Character concept images were first created using GPT Image and subsequently converted into 3D models through the Tripo API. A customized version of UniRig was then employed to automate the rigging process, preparing the models for use within the Unity environment. In addition, GPT Image and Seedance were used to create animated sequences for narrative introductions and scene transitions. The folding-screen image generation system operates based on the visitor’s facial information and narrative outcome. A pre-constructed Korean painting-style template serves as the visual foundation, while the visitor’s facial characteristics and story progression are incorporated into a newly generated folding-screen scene. A ComfyUI-based generation pipeline utilizing SDXL, LoRA, ControlNet, and IP-Adapter was developed for this purpose. Generated images are automatically stored in cloud storage and displayed in real time through TouchDesigner on the digital folding-screen installation. The results are simultaneously archived on a web platform, allowing visitors to revisit their experience after the exhibition.

System Architecture

최종.png

_The diagram shows the workflow from visitor setup to interaction, image generation, projection mapping, and web archiving._

Key Technologies

  • XR / Interaction: Unity, Azure Kinect SDK
  • Action Recognition: EfficientGCN-B0, ONNX, Unity Inference Engine
  • AI Content Creation: GPT Image, Tripo API, Customized UniRig, Seedance
  • Image Generation: SDXL, LoRA, ControlNet, IP-Adapter, ComfyUI
  • Visualization: TouchDesigner
  • Cloud Infrastructure: Cloudflare R2, Cloudflare D1
  • Web Archive: Next.js

Exhibition Images

KakaoTalk_20260706_225943419.png
PROJECTS LIST