SCALE provides a framework of processing load data from distributed load-sensitive modules for exploring force-based interaction. Force conveys not only the force vector itself but also rich information of activities, including way of touch, object location and body motion. Our system captures these interactions on a single pipeline of load data processing. Furthermore, we have expanded the interaction area from a flat 2D surface to 3D volume by building a mathematical framework, which enables us to capture the vertical height of a touch point. This technical invention opens broad applications, including general shape capturing and remote pointing.
We have packaged the framework into a physical prototyping kit, and conducted a workshop with product designers to evaluate our system in practical scenarios.
[Full paper accepted by UIST2019]
Yoshida, T., Shen, X., & Ishii, H. SCALE: Enhancing Force-based Interaction by Processing Load Data from Load Sensitive Modules. ACM UIST2019.
SCALE (2018) is a load sensitive surface which provides exact contact position in 3D and a force vector of user’s touch from low cost load sensors without any prior knowledge about shape of objects. Load based approach is advantageous since they provide a force vector, however, there have been mathematical ambiguity of the touch position along the line of action.
To address this problem, we develop algorithm based on pseudoinverse matrix framework, utilizing the notion that user’s hands are always fluctuating slightly and unconsciously during touch interaction. In this paper, we introduce the design space of the surface, outline of the algorithm and highlighted applications. This technique to localize the touch position in physical spaces enables us to design tangible interaction with inert everyday objects and analyze user’s activities happening on the surfaces
Mengxi Tan & Xiaoyan Shen
Losing a loved one is one the most painful experience that almost every one of us will have to feel some time in our lives. It is never been an easy thing, and there is no efficient band-aid that will cure it since it is not a tangible, physical scar. Research indicates that bereaved people are having a higher risk of serious physical and mental health problem. It increases the vulnerability of some individuals in grief. In this project, we propose an incentive system capable of recording the wishes of children and young adults with terminal disease, and ultimately transforming their wishes into actionable items for their bereaved parents to fight depression.
Yoshida,T., Shen, X., Yoshino, K., Ishii, H.
Toward efficient supply chains and comfortable shopping experiences in physical stores, retail-tech solutions are expected to expand into small stores in rising trend, in addition to conventional large chain stores. However, these solutions have difficulty in deploying to smaller stores including farmer's market. To address this issue, we propose a new goods control system, KI/OSK. The simple table-top sensing system is developed utilizing just three load sensors for helping customer and seller to reduce their operations in-store by grasping goods handling behaviors including picking-up, putting-on and touching, and to install it easily. This project aims to take consideration of the findings in order to explore future retail solutions with HCI technology.
*Paper submitted to DIS 2019, under review.
Mole Messenger is a pair of connected creatures that help children share and send messages to their loved ones who may be far away. Each Mole Messenger box houses a pushable mole. When one is pushed, the other pops up, as if traversing to the other side of the globe. The mole can be tapped to carry different color messages when pushed and can be used to play physically engaged games across distances. This paper describes the design and implementation of the system, explores various approaches within the design space, engages with the context and user experience, and meditates on the potential for positive psychological impact. Inspired by one family’s story of separation and how they helped their children stay connected, we hope Mole Messenger can become an important tool for communication and healthy emotional development in children.
Shen, X., George, M., Hernandez, S., Park, A., Liu, Y., & Ishii, H., “Mole Messenger: Pushable Interfaces for Connecting Family at A Distance”, ACM TEI’2019 (2019.3)
10 x 10 LED installation.
Konrad Feiler & Rikas Shen
Our concept is to provide a self administered, entertaining and approachable method of relief from depression to patients with diabetes. We propose a lush tranquil exploratory virtual environment with moments of discovery, gratification and reward. Our experience will allow the user to freely explore a lush natural setting with moments of magical discovery, distracting the user’s attention from their illness and depression - allowing them moments of joy that can actually improve their condition and quality of life.
Rikas Shen, Sam Liao, and Sauman Tse
Twenty water samples were collected from the seas around Antarctica, Argentina and Hong Kong, recording the exact time and GPS location of each sample as well as environmental parameters like water temperature, salinity and turbidity. Through an optical microscope, photos and videos of diverse phytoplankton and protozoa were documented and then analyzed through their DNA sequences.
The artwork uses a 5-meter dome as an immersive environment. The images of the microscopic organisms are re-animated to reconstruct the tiniest group in the massive ecosystem. The organisms’ DNA sequences of varying nucleotides are also transformed into musical notes and chords. The animation and music combine in a melody of micro-organisms.
Freeze Frame, Extreme Environments: Antactica Exhibition, School of Creative Media, City University of Hong Kong, Hong Kong.
Wen Wei Po
South China Morning Post
Resonance Ver.M is an interactive video projection installation that allows the audience to have a peep into the artist’s inside world. The project is the combination of a series of experimental executions including self-training, performance, EEG recording, interpretation of the bio-signal, subjective dream log, exhibition and interaction. The artwork shows an approach to the investigation of the dreams and a new form of “Human-Human interaction” through electrophysiological signals. It also introduces an innovative form of interaction between the reality and dream world, the conscious mind and the unconscious brain.
Shen, X., George, M., Hernandez, S., Park, A., Mazzola, C. & Ishii, H.
Starscope is a magic staff that allows the holder to hear real sounds of stars and other astronomical bodies in the sky. Starscope began as a project to help our visually impaired friends participate in stargazing and hobbyist astronomy, however, it has proved to be a fantastic tool for sighted stargazers as well. The Starscope uses GPS, a gyroscope, and star location data to create a “hit field” of sound around important stars, so that users can find and pinpoint the appropriate sector of the sky by the strength of the sound emitted and the vibration of the staff. Starscope draws on data from astrophysics research, including asteroseismological recordings and light graphs converted to audio, providing the user access to the sounds of the universe. Some space sounds can get a bit scary, so there is also a “musical” setting of delicate notes and chords. Starscope reveals the symphony of the night sky.
2018 - Programmable Keyboard Workshop Speaker
Organize the event and design courses for people to learn basic information and knowledge about electronic wiring, e-textile, wearable, prototyping and Arduino programming.
Rikas Shen, AnnaLi, and Joshua Cui
Colorful Song is an experimental instrument combining the act of painting with music performance. Artists can create and modulate sound by applying watercolor onto the canvas. Data collected from moisture sensors embedded underneath are fed into Max/MSP to map into different sounds. The work is an attempt to discover and develop innovative and future forms of music performance.
Performed at Jockey Club Street Music Series, Hong Kong Art Centre, Hong Kong.
Rikas Shen, Lucien Lau and Jessie Ye
Blossom Dress is a transforming dress that reacts to data collected by the bending sensors embedded at the elbow and back. When the user feels unconfident and holds themself back, the dress appears like a pale bud. If the user straightens her posture to show her confidence, the bending sensors will control servomotors and the dress will gradually expand, creating an effect like a flower blossom. Meanwhile, a pulse sensor will monitor changes in the user's heart rate to cause the RGB LED strip to blink and change colors.
The Blossom Dress is designed in layers and embedded with a structure similar to an umbrella frame. Additionally, there are servomotors on both sides of the dress that control the expansion of the dress. Attached to the servomotors are two lightweight plastic strips that lift the fabric as the dress expands.
* The project is kept and archived in CityU Wearable Innovation Lab.
#Bio-art #Antibiotic #Processing #Genetic Programming #Installation #Projection #Documentary
Uncertain Blackbox is an installation combining biomedical experiment with genetic programming to explore the impact of antibiotics on human life and society.
Two main "inputs" and "outputs" are presented:
1. Input: Eight samples of chicken breast imported from different countries were bought and residues of fluoroquinolones were tested.
Output: The projected animation is a "micro-ecosystem" programmed based on the data collected from lab experiment.
2. Input: Selected scientific papers describing four studies on antibiotics
Output: Exaggerated/ distorted summary of the paper contents
The rifts between the inputs and outputs are trying to call on an introspection of paying more attention to the black box between the start and the end.
Exhibited in SCM Annual 2015: Flux, School of Creative Media, Hong Kong.
One of the eight Selected Projects (out of 63 exhibited works).
Refractive index (RI) is a fundamental optical property of materials. It is a dimensionless number that describes how fast light propagates through the material compare to the speed of light in vacuum. The light will be bent if it runs into a material with a different RI, and the degree of the angle of incidence is determined by the RI of the materials. If the light enters a material with the same RI, the light does not bend, and no edge is detected. In other words, the material becomes invisible.
Many index-matching experiments glass (n~1.474) is always used since its RI is close to the RI of Glycerin (n~1.473). However, glass is not an ideal material for prototyping customized installation structures, thus I wonder if it is possible to create a mixture of solution which can match the refractive index of Acrylic (n~1.490 - 1.492). At the beginning I thought it could be a very interesting and powerful tool for many different applications.
After some research and experiments, the special chemical mixture of methyl salicylate (n~1.538) and isopropyl alcohol (n~1.478) is created. Although the solution is able to match the RI of clear acrylic. This chemical is a strong solvent for PMMA so the mixture keeps dissolving the solid acrylic and it can destroy a structure in 1 – 10 hours based on the structure of acrylic. This characteristic makes the project an attribute of transiency. The intensive dynamic reaction is happening all the time while nothing is sensed by our photoreceptors. It is there, it is collapsing, it is dissolved—— it exists, but it can never be sensed.
So currently I am further experimenting on other plastics such as polypropylene, which can be decomposed visibly. It is still an ongoing artistic experiment on shapes, materials and dynamics.