Custom Software Development and Engineering

OnScreen keyboard implemented in HTML5 canvas with multitouch support via touch events API and Magictouch.js

Note: Just want to see the goods? the demo, the video.

We’ve recently been building a few applications for PQ Labs multitouch overlays. It’s a very sweet piece of hardware offering 12 simultaneous touch points, a solid SDK, big sizes, and TUIO support.

While a native SDK is nice, if there’s one thing we love it’s the open web. We set out to start writing a few web apps that could be run either locally from the connected machine or from the web that would offer the ability to use the multitouch aspects of the overlay in a web browser.

One particular use case (soon to be tested in the field) was an event based guest book. Walk up to the screen, enter your details, (yada yada more features yada), and away you go. Stores the data via localStorage (no internet access required for the device) and can be dumped for review later on.

The problem with this idea in a browser was that the onscreen keyboard of choice just doesn’t work as seamless as we’d like. It’ll hide itself when not in use, you can lose it behind the browser if you touch focus away, you can’t have three of them in use at once. So, we decided to roll our own onscreen, in browser keyboard that supports multitouch.

To do this via desktop Chrome or Firefox, you need TUIO support. Just so happens there is a npTuioClient NPAPI plugin that offers the very support that we need. For us, the testing is easy because we have the big ‘ol overlay to work with. But if you want to test our demo on your desktop sans the overlay, Boris Smus wrote a sweet article on the topic you should read: Multi-Touch for your Desktop Browser.

Which gets us to the nuts and bolts. The onscreen keyboard uses a couple different things. One, it uses the touch events API, and two, it use’s Smus’ MagicTouch.js polyfill to convert the npTuioClient callbacks to the compatible touch events as defined by the touch events API.

Once that’s up and working, we can now track multiple touches on the screen, which makes things like the shift key work really nice (as seen in the video below), but also allows us to run multiple keyboards on the same screen (which is for an HTML5 game we’re doing a little something something on…that’s a video for another day :-).

The demo on the lab page is the most basic version. The keyboard is defined via an object (you could technically load it up from a JSON file very easy like), the layout can be defined at the keyboard level or set to a global default, the click sound is just an HTML5 audio element. We designed it with flexibility in mind.

As for the touch support, you can sorta run the lab demo on an iPad or Xoom/Tab, but it’s laggy with the full keyboard set (not to mention the wrong size for those screens). That’s really not what it’s designed for, but could be tweaked to make it work.

If you don’t want to setup the emulation of multitouch on your desktop or don’t want to fire up your overlay, than have a peak at the video below to see it in action in Google Chrome: YouTube demo video.

Ready to start?

Get in touch. We're ready to listen.