So here's a crazy idea - what if you played computer games without controllers? What if you could cast spells by making arcane hand movements, just like a real wizard?
It turns out you can. All you need is to hold a strongly-colored object and some basic computer vision, and suddenly you've effectively got a magic wand.
I worked on getting the color tracking working, and on recording the symbols drawn by users (saving them as images). I also hooked up the webcam computer vision-based controller to the main socket server
The main idea here is to have "wands" which are distinctly colored objects (that are different enough from your surroundings for color tracking). The python script uses OpenCV in order to look at the webcam input, figure out where these objects are, and let you start drawing patterns and symbols.
In order to interpret the symbols and translate them into spells, funnily enough, we decided to use MNIST (the popular handwritten digit datasets). So the process is that you draw a digit, and the digit gets translated into a spell, sent over the socket server to a computer that is running linux and emulating Xbox contollers. This computer then emulates the right combination of key presses to cast the spell. Easy :)
One additional thing was 360 degree character movement. We did this by fixing an anchor point that you would move your left hand relative to. Flip your hand to reveal the color to the webcam and that spatial point is your anchor. Move your color relative to that, and your character moves in that relative direction.