/mceVideoSearch

Embedded WinForm Controls hosted in MCE

http://www.brains-N-brawn.com/mceVideoSearch 2/13/2006 casey chesnut

comment(s) 

Introduction

the last article /mceDivX360 left me wanting an MCE programming model that would allow for a rich UI (without having to resort to Hosted HTML or waiting for Vista). as i was nosing my way around the developer.mediacenter newsgroup i came across a post that caught my attention : .NET Applets in MCE. for some reason i did not think this was possible and that only ActiveX controls would work? but Dean, of 10footgames.net, linked to some sample code that did just that. even better, the mcepeaks v2 game is developed using that model and it even works on an XBox 360 extender! immediately started looking around for more sample code and info on this topic ... but there wasn't much to find. but Dean did point me to DotNETris which provides source code and he also answered the many questions i had based on his experiences with mcepeaksv2. so a # of the tips outlined below were originally discovered by Dean.

this article will explore the Embedded WinForm Control programming model which gives us rich UI capabilities and a powerful application model. the end result of the article is an application which uses Yahoo web services to do video search and playback on MCE (and the XBox 360).

Hosted HTML

Hosted HTML is the SDK approved way to make full screen UI apps for MCE 2005. the problem is MCE does not provide a web server. to get around this people have done many things :

  1. have a windows service running that generates static HTML
  2. install an open source web server such as Cassini to only server local pages
  3. install full blown IIS

ouch, ouch, ouch ... that's more pain than i want for me as a developer or for end users. not to mention, HTML is a pain to develop with and looks like crap on a big screen TV. the only real advantage it offers (other than deployment) is its ability to flow layout and scale. NOTE AJAX does not solve these problems for HTML on TV. the solutions ultimately are WPF (Windows Presentation Foundation) and MCML (Media Center Markup Language) to be released with Vista

until then, this app does require a single Hosted HTML page. it is a static HTML page which is just served from the file system (no web server required). its main purpose is to be the entry point for the app and to host the Embedded WinForm Control in an <object/> tag. it also has some minimal javascript functions to let MCE know this is an MCE enabled page, pass the MediaCenter object to the control, pass remote control commands to the control, init and resize the control, and cleanup. NOTE you can pass parameters to the control from HTML using <param/> tags within <object/> that map to public properties on the Control. another trick you might pull off is to have some javascript function that keeps polling and checking a public property on the Control to trigger some dynamic HTML when the property changes to some value.  not sure if it can support more elaborate events from the control without a security problem?

Embedded WinForm Control

developing the Control is pretty much standard WinForms development. but first the control has to be made to look like an ActiveX control. i did this by starting from an existing code template. all i did was port it over to C# and then create a unique GUID for the control. this GUID then has to be updated in the host HTML page so that it can find the control to host. think this is necessary because of how the page and control are being loaded from the file system.

being loaded from the file system gets rid of some of the security restrictions that embedded controls typically have. so far i've used it to read and write files, read from the registry, etc.. haven't tried it but i'm assuming that IsolatedStorage works too. unlike internet served controls, i've also used it to call remote web servers (from which it was not served). i'm assuming that if you run the control in an extender session, that it will reduce security, but i haven't bumped into any of these limitations yet.

the following sections will go into more detail about developing the WinForm Control

MediaCenter Object

when MCE loads a Hosted HTML app, it creates a separate process EhExtHost.exe for it to run in. this process loads .NET 1.1, gets access to a MediaCenter object, spins up some flavor of IE to host the HTML, and passes the MediaCenter object into IE so that the page can get access to it. NOTE you cannot use .NET 2.0 until Vista. you'll probably want to pass the MediaCenter object into the Control so that you can use it to display dialogs, play media, etc... the problem is that the Control will not get access to type information (MediaCenter.Extensibility.MCMediaCenter). instead it will just have to treat it as an untyped object. this really sucks because C# cannot do late binding. if you want to make a late binding call in C# it will look something like this :

object[] args = new object[] { text, caption, (int) buttons, timeout, modal };
object retLate = LateBinding.LateGet(mce, null, "Dialog", args, null, null);
retVal = (DialogDismissed) retLate;

trust me ... to get that to work involved alot of runtime debugging. and the LateBinding type is one of those 'use at your own risk' classes. believe it or not ... VB.NET works better in this scenario because it does support late binding (CAN"T BELIEVE I JUST TYPED THAT). so i created a VB.NET class library to handle all the late bound calls to the Media Center object. NOTE you still don't get intellisense, but at least it compiles and runs. now the above code looks like this :

Dialog = mce.Dialog(text, caption, buttons, timeout, modal)';

er, um ... just starting my 6th year of .NET development and i believe this'll be the 1st time ever that i've released any VB.NET source code ... say it ain't so. hopefully we can get a typed instance of this object in Vista. now that we can call the MediaCenter object ... what can it do? it has 3 different ways to bring up a dialog (Dialog, DialogEx, DialogNotification). you'll want to use one of these instead of MessageBox. you can get the return value to see which button was pressed, but i have not been able to get the callback event to work. it can also NavigateToPage() and GoToFullScreen(). then there is ReturnToApplication() and CloseApplication(). even mess with the shared viewport. the pic below shows my sandbox app for testing, in the upper right its displaying a DialogNotification that was created from the Control

there is more that the MediaCenter object provides, but i mainly need it for the PlayMediaEx() call. PlayMediaEx can be used to play audio, video, etc... and you can specify whether to add it to the queue or play immediately. works great ... most of the time. the problem is if you are trying to make it play a file on the internet. if you pass it a video stream that takes 5 minutes to buffer, then the call will not return for 5 minutes. you will just see the MCE hourglass minute for 5 minutes and your app will be locked that entire time. as far as i can tell, once you call PlayMediaEx, you cannot take it back until the media plays or it times out. even worse, you cannot reduce the timeout. this sucks! this needs to be fixed in Vista. if it is taking too long for some media to start playing, then the user has to be allowed to cancel out. NOTE because of my limited MCE setup, i could only test this on the XBox 360 Extender. i'm not positive if the behavior is exactly the same directly on an MCE PC?

finally, i tried to use the MediaState API from the Control. it should work, but the MediaState API is still a 2nd class citizen. although, it is now signed with Update Rollup 2 it is still in a 'Samples' namespace and you have to run registry scripts to get it to work. hopefully the MediaState API will be fully supported in Vista.

Input

now that we can make the Control do all sorts of cool media stuff, the user needs to be able to actually control it. the primary input will be remote control commands. the hosting page will have an onRemoteEvent that can call a public method in the control passing the key press info. for my app i just used MCEController to then send the corresponding keyboard commands. er, um ... think i've used the MCEController code in every MCE application i've written. make that another request for the Vista SDK.

sending the keyboard commands allowed me to use the remote arrow keys to change focus among buttons and for using the Ok button on the remote to click the buttons on the Control. to change focus with a TextBox i had to subclass it and override the KeyDown event. in the event, it looks for the arrow keys and will call SelectNextControl() to change focus to sibling controls.

then, i went and added some more keyboard and mouse click events to the app so that it could also be used directly on an MCE PC.  so now it works with mouse, keyboard, and remote on MCE PC. it also works with the wireless controller and an XBox 360. best of all you can use the 'Y' button on the XBox 360 controller to bring up the XBox 360 popup keyboard to enter text into the subclassed TextBox (pictured below. MCE needs a generic soft keyboard like this too). to open the soft keyboard using an XBox 360 remote control press the 'DVD Menu' button (not the yellow 'Y' button). NOTE if there is existing text in the TextBox you cant add it to the popup keyboard to be edited. NOTE was not able to test this with a USB keyboard on the XBox 360 ... so i'm not sure if it could input into the TextBox on the Control?

finally, only being able to enter text with an MCE remote control (directly on an MCE PC) was killing me ... so i implemented Triple-tap functionality. all i did was add a reference to the existing Triple-tap Control that comes with the MCE SDK. now if you click the OK button on the remote, it will bring up the Triple-tap control. then you can use the # buttons on the remote to enter text. then use one of the arrow keys to transfer out of the Triple-tap control.

Triple-tap works ok directly on an MCE PC, but not great. and its really hard to use over an XBox 360 Extender. the reason is the display doesn't refresh fast enough to let you know what you entered so its real easy to screw up. but it just so happens that the Triple-tap control has a SoftKey mode as well (pictured below). but for the life of me, i could not get it to switch into that mode. the control was meant to be used in HTML and not a WinForms control ... so i never could get it to switch. regardless, the XBox 360 soft keyboard is much better. this make me wish for 2 things : 1) that MCE PC would get a generic popup soft keyboard or T9 that was accessible by all applications. such that you could press some button on the remote to bring it up and it would behave the same as the 360s popup keyboard. 2) or that our apps could determine that it was running in an extender session and send some remote command to popup the XBox 360s keyboard. so overall, i'm pretty unhappy with the text entry capabilities for MCE ... Vista needs to improve this. the one good thing is that i did buy up the MCE remote keyboard and like it alot ... just needs to work with the 360 too

Focus

this becomes a problem now that the user can move around the screen. i started out relying on TabOrder. so the user clicks the arrow keys and the next or previous control is selected based on tab order. the left and up arrow keys select the previous control and the right and down arrows select the next control. to keep things simple i made it a chain. once you get to the last control then it loops back around to the first. not great, but it was simple. the only real trick was having to subclass the TextBox so that it would relinquish focus once it had captured it. NOTE i've got a bug when clicking off of the TextBox that changes focus to the control 2 clicks away ... haven't been able to track this down yet? the other trick was to give feedback as to which control was selected. did this by creating a timer. on the timer tick it checks every control in the controls collection to find the focused item. if the item is focused then it sets its background color to Gray.

the TabOrder navigation technique (above) was just too limiting, so i reimplemented it to work as expected. so now if you press the right arrow, then it will move to the control to the right and not just the next control according to TabOrder. this is done with some basic bath. whenever the controls are scaled to fit on the page, their centerpoints are saved to a collection based on their Location and Size. then when a right click comes in, the distance and slope to each control is calculated. then the closest control to the right is selected and Focus is called on it. this makes navigation much more intuitive.

Sound

sound effects are good. when running the control on MCE you can call VBs Beep, PInvoke PlaySound, use SAPI for speech synthesis, and of course PlayMedia. on the HTML side you can use the <bgsound/> tag. the problem is that only PlayMedia works on an Extender ... and it's too heavy for short sound effects. Remote Desktop does allow for outgoing sounds (not incoming) so i wish they would make the Extenders support it. and since Remote Desktop does not support incoming sounds, that means the app can't do speech recognition from an extender :(

for this app, i added some basic sound effects that will be played when run directly on an MCE PC. if on an Extender, the sound effects wont be played, so i added some additional visual cues that help with usability. hopefully sound effects will be supported by Extenders in the future.

Render

this is the money shot. this model provides all the graphical richness of WinForms and even lets you drop down to GDI+. you can seriously use this model to write a GDI+ game running at 30+ fps and would not notice any lag directly on the MCE PC. NOTE an extender only gets like 3 to 5 frames per second, so its not adequate for twitch gaming. on my fully wireless setup, i can even notice the slight lag from DotNETris when running it in an extender session ... but it plays perfect directly on the MCE PC. whereas the card game mcepeaksv2 plays great on both the Extender and MCE PC.

for my app i tried to stick with the existing controls as much as possible. still, i had to subclass some controls to slightly modify their behavior, and ultimately i resorted to lower level GDI+ for parts of it as well. the next step would be to introduce some animations.

to be clear, the app refreshes just like a normal app on MCE PC with no lag. you only get the 3 to 5 fps refresh rate to an Extender because its running over remote desktop. but that doesn't mean that the video runs at 3 to 5 fps ... the video still plays normally on an extender because it does not go over the remote desktop connection. the video is buffered, scaled, and played directly on the XBox 360.

Scaling

the only trick with rendering is having to scale everything. when the hosting page is loaded it gets the size of the display which can be passed into the control. on my widescreen notebook its 1400x900, while on an old TV its something like 600x450. the app needs to be able to scale to fit both these displays and more. when the size is passed into the control then i resize the control to take up the whole display, then the control calls Resize on itself. the Resize event then does basic math to place objects on the screen and modify their size (e.g. the buttons should only take up 1/5th of the screen width). although you could use WinForm docking and anchoring to help, my app entirely ignores the design time surface. it also modifies the size of the font so that the text will scale for large displays. in this manner you can tailor the app for standard or widescreen televisions. e.g. for standard TVs it shows 3 columns of thumbnails, but it could be made to display 4 columns with the extra space on a widescreen TV. the pics below show the same search scaled for standard TV and widescreen TV.

 

Improvement Requests

here's a wrapup of improvements i would like for both Vista MCE and the XBox 360 Extender

now that the programming model had been explored, it was time to put it to the test by actually writing something.

Video Search

since i don't have a TV tuner on my MCE notebook and my video card only has 32 megs of RAM, which means it cant even play video through MCE, then the XBox 360 is the only way i can watch video content from MCE. the problem is i didn't want to manually collect all of those videos beforehand to add them to 'My Videos'. what i wanted to do is get access to the video content that is served from the web. thus MCE video search. so this app will collect your search term and find videos using Yahoo's Video Search Web Service. then it will display those results with thumbnails and metadata. it also allows you to page through the search results or select a random page. when you find a video you want to watch then you just have to select it. this will call PlayMediaEx to play the video on your MCE PC or through the XBox 360 Extender.

here is how to control it with an MCE remote (directly on an MCE PC) :

here is how to control it with an XBox 360 remote control :

here is how to control it with the XBox 360 wireless controller :

you can also use the MCE remote keyboard on an MCE PC : arrow keys, spacebar, and backspace button

the plan was to also make this a sort of video DJ. where you would just type a search term and it would queue up media files from the search results and just play through the entire list. but i didn't add this feature based on the problem with PlayMediaEx(). because of this problem the app can get stuck. e.g. if you click on an 80 meg file that only buffers about 1% a minute, then the app will stay locked. i've been having to click the guide button and end the extender session when this happens. to help get around this a little, the app pings all video files and only calls PlayMediaEx for the files that return HttpStatus OK. this helps the app lock up less ... but not completely. after that it attempts to use the MediaPlayer SDK to play the file. if that fails, then it does not attempt to play it with MCE. next, it starts to download the video behind the scenes to get an estimate of how long buffering could take. if it is really slow, then it will prompt the user if they still want to play the video and possibly have to wait while it downloads. the problem with this logic is it has false positives. sometimes it will start playing immediately after just buffering a small portion of the video. you've been warned. once again, i think it is critical for MS to fix this in Vista ... there's this little thing called the internet ... and its not a friendly place ...

here are the places where it can fail :

Yahoo Web Services

right now it just uses the Yahoo Web Services for Search. specifically it uses the unsupported CSharp wrapper from Martin G. Brown. this adds just a little typing to Yahoo's REST APIs ... man i hate REST. the API lets you specify a search term, media type, adult content flag, etc... these parameters are exposed in the hosting HTML page as <object/><params/> so you can change those as you see fit. now if the adult flag would return adult content only ... that would be cool :) right now the media type is for msmedia only. you can also set it to include mpeg content, out of which some of it should play on the XBox. what's bad is that not all of the msmedia files play on the XBox. i'm thinking that the XBox should support at least all video formats that Yahoo thinks belongs to MS :). the adult flag is set to true ... what did you expect me to test with :). you can change this setting in the file : C:\Program Files\brains-N-brawn\mceVideoSearch\mceVideoSearch.htm (change to <PARAM NAME= "AdultOk" VALUE= "false"/>). the web service also has a limit upon search results. even if there are 9999 videos of janet jackson at the superbowl, the service will only allow you to access the first 4000. finally, the service requires an AppId which it uses to track usage. if i read correctly an AppId can make 5000 requests per day to each unique IP address. so that isn't likely to be a problem ... if it is, then you can apply for your own AppId and use it instead. other than hating REST ... the service is responsive and seems to work great. it even has a decent amount of videos. for some of the searches i've tried repeatedly in the last couple days, it looks like some of those have gotten new content in just the handful of days i've messed around with it! the only problem i had was when trying to retrieve thumbnails. it looks like there server is separating the HTTP Header and Body with \n\n instead of \r\n\r\n.

Video

the first video shows it running in an XBox 360 extender session using the wireless controller and Soft Keyboard

 xbox360_wireless.zip

the second video shows it running on my MCE Notebook using a remote control and Triple-Tap. NOTE i couldn't play a sample video in this one because my MCE Notebooks video card is too small for video playback through MCE; it can only do images and video ... that is why i have to make everything work with the XBox 360 :) this definitely adds more worth to the people that bought an MCE PC without a TV tuner.

 mcePC_remote.zip

Conclusion

this article explored how to write Embedded WinForms Controls for Media Center. this provides us with a powerful development experience and allows us to control the entire UI to a level that is not possible with Hosted HTML. kind of surprised that this model was not adopted and included in the Media Center SDK. to demo the programming model, an MCE video search application was created which can be used through the XBox 360. it works ok, except for whenever a call to PlayMediaEx() locks up :(

if somebody felt like it they could make this code more generic (i.e. interfaces) to support other video search services (e.g. google). does MSN have a video search service yet? ... maybe with Live? or you could change it into an MCE music search for audio (or images). finally, you could add a backend server with ratings to basically create a digg.com specific for media. this same technique is begging to be applied to /mceDivX360 or Transcode360. then we could use it to display thumbnails throughout the video and allow a user to specify where in the video they wanted to start streaming.

Source

here is the source code. mostly C# with a little bit of VB.NET thrown in for late binding :)

here is the installer. if you want to run it on your XBox 360 then make sure you select to install it for 'Everyone' so that its available to the Extender user account

Updates

will only update this if PlayMediaEx() is fixed for Vista

Future

supposed to write something with WWF someday. later