Dan Liebling and Meredith R. Morris from Microsoft Research will be presenting today at the 2012 ACM International Conference on Interactive Tabletops and Surfaces about their project “Kinected Browser: Depth Camera Interaction for the Web“, a toolkit which enables web developers to detect Kinect gestures and speech using a simple JavaScript library.

With the introduction of Kinect for Windows earlier this year and the increasing adoption of HTML5 and CSS3 browsers which support rich multi-touch events, Dan and Meredith saw an opportunity to expand the developer reach of the Kinect SDK beyond its C# and C++ roots.

Depth-camera based gestures can facilitate interaction with the Web on keyboard-and-mouse-free and/or multi-user technologies, such as large display walls or TV sets. We present a toolkit for bringing such gesture affordances into modern Web browsers using existing Web programming methods. Our framework is designed to enable Web programmers to incrementally add this capability with minimum effort by leveraging Web standard DOM structures and event models.

The team recognizes some existing work by the community of Kinect and browser integrations: Swimbrowser, an app that uses Kinect to “swim” around websites but does not offer any developer endpoints, and DepthJS, a Safari and Chrome plugin that exposes a similar JavaScript library with more focus on high-level gestures.

In contrast to these, Kinected Browser is a low-level approach that only exposes raw sensor (RGB and depth) data. It works by two components, a browser plugin that talks directly Kinect C++ SDK and a JavaScript library which a web developer can include and redistribute on the page. In the future, the browser plugin can also be abstracted to work with a number of depth camera systems.

The benefits of this “versatile” approach is that they support many more exciting scenarios such as a “Web applications using depth data, e.g., to scan physical artifacts and input them into 3D printing services” or even multi-user interactions on a single website enabled by facial recognition and skeleton tracking.

Compared to Xbox game titles or apps, distributing and promoting a website is much easier. It would be fascinating to see if websites becomes a popular platform for new Kinect hacks and games.

The Kinected Browser library should be available for download from the Microsoft Research project site later today.

Just as I was leaving Seattle at the conclusion of Microsoft BUILD 2012, a HTC 8X fell off the back of a truck and into my hands. I’ve been keeping it in my pocket over the last couple of days, doubling my dosage of radiation with the iPhone 5, and it’s a Windows Phone 8 phone with a lot of potential.

Hardware

It’s hard to single out any one phone from the Windows Phone 8 launch lineup – they’re all quite impressive with a slew of pros and cons. Last week I played with the Nokia Lumia 920, the Nokia Lumia 820, the HTC 8S (briefly) and of course the HTC 8X.

For the most part, most people will be comparing two flagship phones as their first Windows Phone 8 device, the Nokia Lumia 920 and the HTC 8X, and it’s not an easy choice by any stretch of the imagination. If only you could morph the two together, but within the laws of physics I prefer the HTC 8X.

One of the major differentiators between the 8X and the 920 is the size and weight, although neither is “optimal” in my opinion. I find the 920 uncomfortable to hold and use in one hand and even the 8X barely makes the cut. Although I can comfortably grasp the 8X in my palm, it’s irritating to stretch a thumb to the far corner of the screen without regrasping.

The next major point of contention, the camera, is not the 8X’s strong point by any stretch of the imagination. You’d have to be living under a very very dark rock to not have heard about the 920’s PureView camera which I found to be near equal to the iPhone 5 in normal conditions and a generation-ahead in low-light.

In contrast, the 8X is fairly typical current-generation mobile camera with the typical tendency to overexpose and mis-calibrate the white balance. THe photos you take with the 8X will still be nice to look at but won’t be accurate most of the time.

Overall, the 8X has very solid build quality with a matte backing which is much more comfortable to grip and the choice of different colours is a nice touch. The screen is rich and bright to show off the new tile accents of WP8 and also takes advantage of the new 16:9 aspect ratio which makes photos, websites and long lists easier to browse.

It would be hard to find fault with the hardware except one gripe I have with the 8X, the hardware buttons. Small in both thickness and depth, the buttons are designed for ants. Arguably the most important button, the power button, is hard to both locate and press.

Windows Phone 8

Windows Phone 8 is a natural evolution over Windows Phone 7. With more under-the-hood changes than end-user features, the new release is much more refined, responsive and practical in pretty much all the same area that made Windows Phone an interesting mobile OS in the first place.

Taking advantage of the dual-core chip in the 8X, multi=core support in WP8’s new shared kernel appears to deliver a much more responsive usage experience with considerably faster app loading, snappier animations and transitions, and butter-smooth scrolling and pivoting in many day-to-day apps like Facebook.

Other nice enhancements like the vector maps which can be cached by entire regions and countries for offline use, downloadable keyboard languages, autocompletion alternatives for any tapped word, over-the-air updates, and SkyDrive backups for SMS text messages are all cherries on top of what is already a very sweet and tasty cake.

The OS also sets the groundwork for many exciting innovations such as DataSense, NFC Tap+Send and digital Wallet which has rudimentary support at launch from carriers and software developers, but has the potential to be key platform differentiators if the required pieces of the puzzle fall into place.

Software ecosystem

Having said all that, some of the most exciting aspect of WP8 are yet to materialize. Even though the OS is designed to be backwards compatible with all WP7 apps, I am still holding my breathe for a plethora of new and updated apps that will take advantage of the new 16:9 aspect ratio, lock screen background, new APIs and capabilities (ex. VOIP).

The apps which are on my radar but yet to materialize at the time of writing are Skype for Windows Phone 8 which will feature extensive integration into the People hub and background connectivity, as well as the OEM-agnostic free Nokia Drive app to substitute the out-of-box (and practically useless) turn-by-turn experience.

Apps which has already been updated to take advantage of WP8 like Facebook for Windows Phone v4.0 is a prime example of how an app with a few tweaks is much more fun to use including but not limited to the lock screen image and full/mini tile support.

Ever since Xbox SmartGlass was announced at E3 2012, I’ve been intrigued at how it works behind the scenes – across games, apps and device platforms. Thanks to a session on the last day of Microsoft BUILD 2012, it turns out to be quite an interesting developer platform, if you can get your hands on it.

As Microsoft Xbox Live’s Neil Black explained, the goal of SmartGlass is to simplify the trend of second-screen experiences for both end-users and developers. The games, TV and movies market is fragmented with companion-like apps from publishers, networks, devices and special once-off apps like the Olympics.

To make SmartGlass available to the devices that people already own (but best on Microsoft devices), they settled on a client cross-platform HTML5, JavaScript and JSON powered solution for Windows, Windows Phone, iOS and Android devices.

Using the Xbox SmartGlass native app as an abstraction layer, it manages the connections to the SmartGlass cloud services and exposes a set of SmartGlass APIs and events (ex. client connected, Xbox media changes, triggered events) and device capabilities (ex. accelerator, gyro, input) to the hosted browser control (IE or WebKit depending on device).

In fact Microsoft even went as far as to host a service proxy for the SmartGlass client apps to connect to third party REST APIs without running into cross-domain scripting issues. A SmartGlass Studio app (not demoed) was also available to simulate a mobile client from the developer machine.

The result is that a developer without much connectivity or even game developer experience could write a SmartGlass client with just a few lines of HTML and JavaScript as if it was a mobile web site. The example of a Poker game SmartGlass mobile client could utilize the phone’s accelerometer for each of the four players to shake the phone for “table-knock” check in the game.

But of course it was all too good to be true. Microsoft is currently restricting the SmartGlass SDK to Xbox title publishers, which are the likes of EA, Activision and ESPN who have a “partner relationship” with them. Microsoft’s Neil also confirmed that indie game developers using the XNA platform do not have access to SmartGlass and support is “not directly on the roadmap”.

As someone who is a strong advocate of Apple’s AirPlay solution which not only natively supports display mirroring but also allows any iOS developer to develop multi-screen apps and games, it’s disappointing Microsoft is trailing behind as a late arrival.

I wish you could hear the collective sigh from the developers in the conference room when the presentation concluded.

As thousands of BUILD attendees unboxed their Surface RT devices, one of the first things they did was to try out how their Windows 8 apps ran. Unfortunately for many of them, including us the MetroTwit developers, we witnessed notable performance issues.

It turns out this is to be expected, especially since Windows RT devices will be flooding the market with relatively low-powered ARM processors and graphics processors (compared to grunty x86 dev machines). It is recommended developers target and test their apps on the lowest common denominator which at the moment seems to be the Surface RT.

In one of the most useful presentations at BUILD 2012, “Performance tips for Windows Store apps using XAML“, the presenter Kiran Kumar gave out many tips that C# XAML app developers can utilise to address several key performance issues that he sees plaguing most Windows 8 applications.

Startup loading resources

One of the first problems that might plague an app’s perceived performance is the startup time stuck on the splash screen. As Kiran explains, a lot happens in the few seconds that occurs after tapping on the Start tile including loading and parsing all the referenced XAML page resources.

Two tools that can help debug exactly what is going on in the background (and two of the ugliest developer tools) are XPerf and Windows Performance Analyser. When their powers combine, they produce a visual timeline of all the loading events that might be unnecessarily bloating the startup process.

A common issue in many apps is loading all of the resource styles inside App.xaml which is convenient from a developer’s perspective since it’s available application-wide, but unfortunately it also means the resources have to be loaded at startup, instead of loading what is absolutely necessary for the page.

Fill rate

A significant factor to the rendering performance is overdraw which is affected by the relatively low fill rate of Windows RT devices which can only draw each pixel on the screen around 3-4 times.

What this means is that if there are a lot of overlapping elements that render on top of another, it may not have enough time to draw the pixel to reach the desired 60 frames per second for a smooth animated interface experience.

Bizarrely and to many developer’s surprise, even the default templates that Microsoft ships with the Visual Studio 2012 project templates (Common\StandardStyles.xaml) are actually not optimised for fill rate.

Specifically the “Standard250x250ItemTemplate” style has multiple unnecessary layers which overlap one another. Removing these and ensuring elements are split by rows and columns will avoid pixels from rendering on top of one another.

Direct Manipulation hardware acceleration

Last but not least, another significant factor of Windows RT app performance is touch since it is the primary form of input for slates. To address this specifically, Microsoft has “Direct Manipulation” which uses hardware acceleration that makes touch panning much better than mouse.

With that in mind, there are a few gotchas that may inadvertently disable or affect Direct Manipulation’s acceleration for panning which includes hooking into “Pointer” events for moving objects and currently, and the ScrollViewer control is the only method to take advantage of Direct Manipulation.

The notes above are just the tip of the iceberg to what Windows 8 C#/XAML app developers should be aware of when making performant applications. As the app ecosystem is still in its infancy, it’s not unexpected that many apps, including my own, are still learning the tricks of the trade.

No doubt as the ecosystem matures, there will be more tips from Microsoft and third parties that will help shape the best practices for developing performant Windows RT applications for low-powered ARM devices.

If the attendance of design-focused sessions at Microsoft’s BUILD 2012 developer conference is any indication, then Windows 8 developers are rightfully paying much more to implementing better design and user experience in their apps. It turns out they may also be rewarded for doing so.

In a session presented by Lora Heiny and Will Tschumy from the Windows design team, they highlighted the significance of “self-evaluating the UX of your app”.

Even the title of the presentation alone is fairly important since the Windows Store certification process does not enforce the UX guidelines. Having said that, they do assess the UX guidelines when picking which apps to feature. This means a poorly designed app could be approved for the Windows Store, but would never be featured.

The best web resources to get started with Windows 8 app user experience are the UX guidelines on design.windows.com and the success stories on Windows Store portal.

In my own experience the scope and depth of documentation available can be quite overwhelming, which is why I usually only look up the guidelines of the specific UI control I’m implementing as I do it, one at a time. This makes it a lot easier to process and useful.

Another tip by the team is to start the design review process early, even before one line of XAML or HTML is written. In the planning stages of the app, a single sentence should encapsulate what the app is “best at” which will help scope the application and inevitable feature trade-off decisions.

This “best of” sentence will also help later on in the process such as in the app review stage to confirm the UI and UX actually delivers the experience set out to do, as well as writing the Windows Store listing description to promote the app.

Some of the other significant recommendations were:

• Only use the wide tile if there is live content – otherwise app should only have square tile
• In-app grids should use size and shape to differentiate content – there’s no rule the grid-structure in the apps have to be implemented in uniform, different blocks can be different sizes and also can be circular
• Top app bar buttons are for navigation and should not be round buttons – contrast to the bottom app bar buttons which should always be round
• In pages or apps where the snap view is inappropriate to display content, it’s acceptable to just display a logo – like the Windows Store app which has no functionality in snap view
• Being a good search charm citizen (both receive and send content) is highly valued

Although the Store team will take many attributes into consideration when shortlisting apps to feature – downloads, ratings, functionality, compliance with the UI guidelines is one factor that might be holding an app back. This is also a difficult one since there’s no quantifiable measure of compliance with the UX guidelines like a checklist even for self-review.

You can watch a recording of the presentation in full on Channel 9 here.

From a personal perspective, I think one of the reasons why MetroTwit for Windows 8 has not been featured yet is the fact that we disagree with the UX guidelines in a couple different areas at our own discretion, especially when prioritising content over form.

On a similar note, my friend Rafael Rivera brings up a good point that even Microsoft’s own inbox apps like “Weather” purposely breaks from the UX guidelines too for functionality (ex. content should not pan both horizontally and vertically, Weather pans horizontally and has a vertical list by hour).

It’s that time of the year again where your favorite Microsoft bloggers come together to cover the most exciting Microsoft conference of the year, BUILD 2012. To help cover the keynotes, we’re doing another group liveblog with Mary Jo Foley, Paul Thurrott, Kip Kniskern, Rafael Rivera and myself, Long Zheng.

Today is day 2 and wrapping it up is the second keynote where we expect the focus to be on Windows Azure and dev tools.

Note: We are using an experimental liveblogging app. If the embed below does not work, try this link.

Continue reading →