test project using an AR framework I've been building for a music project. The framework can integrate human skeletal pose tracking and ARKit face blend shapes from various sensors into a single pose + face datastream

the view is a point cloud combining data from a Kinect depth camera, and a Unity game engine depth camera, rendering both real-world and game engine objects (the eyeballs) in a unified view with correct occlusion etc

the boxes are triggers in the game engine space which send MIDI notes to a DAW when triggered with the correct joints or gestures from the pose model, or send MIDI continuous controller messages based on the position of the pose-model joints within the trigger volumes (e.g. the pitch bends as i move my hands through the boxes)

the point cloud uses a vertex shader to displace the points in defined areas for the shaking and vibration effects, the interface is like a synth with ADSR envelopes and waveform selection for each effect, triggered by MIDI messages from the gesture tracking system

vtube character controlled by my movements and facial expressions and some foot controllers, based on human nervous system. Built after a discouraging visit with my neurologist. Not really visible in this video, but when any part of the body moves (fingers, arms, legs, facial features), blue signals travel from the brain down the nerves to that body part