Using DirectShow with Media Center Edition
er, um ... i have written a # of MCE articles, but still felt like i was missing an important piece. then it dawned on me that most of my MCE development was UI and/or Control related. those are certainly important pieces to the user when they are at the 10' experience and are only armed with a remote control. but what was missing? ... Media! yep, working with Media would probably be a good idea :) this article will use DirectShow to show how you can use FM and TV media in your own applications. the samples below are built-in a stand-alone WinForms application, but could easily be pushed into MCE (especially Vista). NOTE this is my first attempt at DirectShow programming, which isn't particularly easy, so i'm probably doing some stuff wrong (e.g. i'm definitely not releasing all COM references) and am far from knowing best practices (you've been warned). the pic below shows the app doing TV playback with dual TV tuners
the MediaCenter SDK is really thin when it comes to working with Media. the main call we have exposed for MCE 2005 is PlayMediaEx ... which i already have issues with (see /mceVideoSearch). but MCE uses DirectShow for media capture and playback, including : DVD, FM, Internet Radio, TV, Video, and Audio. Video and Audio playback might actually use MediaPlayer, but MediaPlayer is built with DirectShow, so its just another layer of abstraction. but if you search the SDK for DirectShow, all it really talks about is how you can create a DirectShow filter for playback of media that MCE doesn't natively support.
Stephen Toub has already used DirectShow for some of his MediaCenter articles
DirectShow is a COM library for capture, editing, and playback of media. Windows Media Player is build on top of it and uses it mostly for playback, while Windows Movie Maker is an example of an app that uses it for capture, editing and playback. it's also used by MediaCenter for working with TV Tuners. DirectShow used to be part of the DirectX SDK, but within the last 6 months it has moved to the Platform SDK. nor does it provide any love for managed developers. the good news is that the open source community has provided a managed wrapper on top of DirectShow called DirectShowNet. there are a couple of other wrappers for DirectShow out there, but this is THE one to use. its 1.3 release wraps all 300+ DShow COM interfaces. it also has many sample applications.
the building blocks for DirectShow are Filters, of which there are 3 types : Source, Transform, and Sink. example sources are video/audio files, web cams, tv tuners, etc. sink filters are on the opposite end and either write media to a file or render the audio or video. in the middle, are transform filters which could do a # of things. if the source is a compressed audio file, then a transform filter would uncompress the audio into a format that a sink filter can play. or if the source is a web cam then a transform filter might compress the video so that a sink filter could capture the video to disk. the filters have pins which can be for input or output. source filters generally only have output pins. sink filters generally have input pins, and transform filters usually have both. the output pin from one filter is then connected to an input pin of another filter. connecting filters together forms a graph, and this is called a FilterGraph. Filters can also have property pages which can be used to configure a filter.
to take a look at an actual FilterGraph, you can use the program GraphEdit. it is installed with the Platform SDK at C:\Program Files\Microsoft Platform SDK\Bin\graphedt.exe. NOTE you might have to register proppage.dll first (regsvr32 proppage.dll). open up GraphEdit, and then click File - Render Media File. select some standard file like an MP3, WMA, or WMV. it will then attempt to build the FilterGraph to render that file. the pic below shows the FilterGraph for a WMV video.
press the play button and it will open up an ActiveMovie window showing the video playback. the source filter in this graph is tied to a WMV file. it has 2 output pins, 1 for audio, and 1 for video. the audio path goes through 2 transform filters to decode the audio, and is finally played using the default DirectSound sink filter. the video path goes through one transform filter and is then rendered using the default Video Renderer. you could also tweak the filter settings. right-clicking on the AC3Filter brings up a property page for settings that can be modified. or you could entirely change the graph. e.g. you could delete the video filters so that playing the graph will only output audio. then you could add filters to convert the windows media audio into mp3.
next click on Graph - Insert Filters. this will bring up a dialog showing all the DirectShow filters that are registered on your machine. these are divided up into categories for audio/video capture, effects, decoders, compressors, etc... but most of the filters i've started out with are in the DirectShow filters category. also, you can find filters for hardware devices you might have like web cams and TV tuners. and this is where i initially had trouble with DirectShow; because it is sometimes hard figuring out which filters can be connected together. but GraphEdit is a great tool for you to experiment with.
you can also use GraphEdit to see what FilterGraphs MCE is using (at least for playback). fire up MCE and start playing RecordedTv. then, in GraphEdit click File - Connect To Remote Graph. this will open up a dialog with at least one item in it. select that item and you should see the FilterGraph that MCE is using to playback RecordedTv. the source filter in this instance will be tied to a dvr-ms file. for LiveTv (one of the pics below), it will be a .TempSBE file (SBE standing for Stream Buffer Engine) which allows you to pause LiveTv.
anyway, these FilterGraphs are visible to GraphEdit because MCE publishes the filter graph to the Running Objects Table (ROT). i'm not exactly sure why MCE does this ... possibly for debugging purposes? i don't know COM, so i don't really know anything about the ROT, except i like it. GraphEdit can get to the filter graphs that MCE uses for playback because it (ehshell.exe) runs under the current user account. for recording, the graphs run in a windows service (ehrecvr.exe) under the LocalSystem account. NOTE it is unknown if Vista MCE will continue to publish its filter graphs to the ROT. i really hope it does.
the following pics are filter graphs from MCE 2005. some of the filters will vary based on what hardware (e.g. tuners, video/sound cards) and filters you have installed.
this shows that GraphEdit is an extremely useful tool for experimenting with DirectShow and also to get a better idea of what MCE is doing behind the scenes. of course you can work with graphs programmatically too. the sample code shows how to load a graph that is in the ROT as well as publishing your own graphs to the ROT. an example of this was used by Stephen Toub for his TimeTravel app. that app loads the graph from the ROT and then finds the source filter to determine what the currently playing dvr-ms files is. this article will discuss some other uses below.
to learn more about programming DirectShow, i recommend reading the DirectShow SDK and the MS Press book (code is in C++) : Programming Microsoft DirectShow for Digital Video and Television. and the C# samples that come with DirectShowNet are also very useful.
the rest of this article only looks at using DirectShow for working with MCE TV Tuners. i wrote this code using 2 different analog USB TV Tuners
both of these cards are analog, USB 2.0, and support FM. the Adaptec is unique because it is a dual tuner. i used basic cable with both of these cards connected by coax. MCE can also work with OTA HD tuners, but i have not purchased one of those yet to work with. to abstract working with analog and digital tuners, MCE uses the Video Control. to get a better idea of what MCE does, you can read Media Center Technical Discussion 1: Tuners and TuningSpaces. the sample code below does not use the Video Control, so i'm assuming it will only work with analog tuners. also, i could not get audio to work with the Adaptec tuner. anyway, i'm not sure how likely this code will work with your own tuner?
one of the first things to overcome was finding an available tuner. this works by querying DirectShow for all the VideoCaptureDevices. the problem with this call is it returns more than just TvTuners (such as web cams). to determine if the device is actually a TvTuner then you have to query for the IAMTVTuner interface. if it's not a TvTuner, then it will exception. the next problem is finding out if the tuner is already in use, because MCE could already be using a tuner. the only way i could determine if the tuner was already being used is to create a filter graph using the tuner. then if you try to run the graph and the tuner is already being used, the graph will exception. if the tuner is not being used then it will not exception. yuck! the key is to create the graph without changing any settings before you try to run it. because if you change a setting for a TvTuner that is in use, then you will effect the program that is using the tuner. e.g. if MCE is recording with a tuner, and you create a graph with that tuner and change the channel, then MCE will end up recording the channel that you changed its setting to. from checking the directshow newsgroup, this was really the best info i could find for working with dual tuners. i'm assuming that using the Video Control would make this easier? if not, i hope DirectShow is updated to make it easier to find an available tuner. also, DirectShow should make it possible to lock a tuner so that another app cannot come and stomp on its settings. er, um ... and although i think DirectShow should make this easier, this is also considered a driver problem. the TvTuner drivers should keep these conflicts from occurring. the bad part is both my tuners are branded as MCE tuners, so i would assume that the drivers have already been certified ...
if there are multiple tuners, then my code tries to find one that is not in use, and it tries to do this without effecting the setting for a tuner that is already in use. the problem is MCE doesn't follow this same pattern. i.e. MCE is not good at sharing tuners. e.g. if i start my app and have it tuned to channel 2, and then MCE starts up to record channel 4, then my app will be changed to channel 4 and MCE will fail because the tuner is already in use. i consider this bad and hope Vista MCE does not exhibit the same behavior. i do understand how MCE should have priority, but there should be at least some means for a 3rd party app to override that. e.g. possibly for a home security app that is plugged into MCE for home automation. supposedly MCE does fire a event before it needs a tuner. checking the MediaState API, there is an event called RequestForTuner. but i tried to catch this event and never caught it. so either i'm doing something wrong or it never gets fired? really wish that my app could hook this event, because then at least it could stop using the tuner. then my app would be hosed, but at least MCE would work correctly ... instead of having them both fail. the current hack i have to minimize this problem is to attempt to use TvTuners in reverse order. in a multi-tuner setup, my app will first attempt to use the tuner that MCE will attempt to use last.
another problem i had was both my tuners have MPEG-2 encoding hardware. almost all of the samples i could find on the web did not use MPEG-2 encoding. what made this difficult is that you cant just attach the MPEG-2 Demultiplexer filter to the graph and connect its pins, because it starts out with no output pins. so you have to create those output pins and map the streams accordingly. when creating the output pins, you have to create MediaTypes to represent both audio and video. this might be where the problem is in which i can't get audio with the Adaptec tuners? anyway, i had problems getting those parameters right, and grabbed that code from MediaPortal. MediaPortal is an open source C# media center. it uses an older DirectShow wrapper library from codeproject.com, so the code isn't exactly the same, but it's easy enough to follow along.
started out using the tuner to play FM. NOTE i could not get audio to work with the Adaptec tuners, so this only worked with the Hauppauge tuner. the resulting graph for playback looks like the following :
for FM, as opposed to TV, it just has to set the tuners Mode to AMTunerModeType.FMRadio. and the station can be set by setting its channel to a radio frequency (e.g. 94.5 would be 94500000). an MCE app could use this capability to provide fine tuning of radio stations.
i also tried to use this code to try and auto tune radio stations. with MCE 2005 you have to manually scan for radio stations and add them as presets. so my app automatically loops through all the radio frequencies and tries to determine if there is a signal present. if a signal is present then the station could automatically be added as a preset. the problem with this is that it sometimes locks on stations that get bad reception, so the user might have to remove some of the auto tuned presets. or some additional audio processing could be done to try and handle this without user intervention. the MCE SDK does not provide an API to store FM presets, but it just writes them to an XML file here : C:\Documents and Settings\[CurrentUser]\Application Data\Microsoft\eHome\ehshell.config. NOTE that extenders store their own FM settings. instead of the CurrentUser directory, it will be something like MCX1.
finally, i extended the graph to record FM to disk. this is because MCE 2005 only lets you pause live radio, but does not allow you to record. because i got lazy, the app records and previews the app in the same graph (remember that MCE does this separately). this doesn't work so great because sometimes the preview will skip, although the recording will still play back fine. the graph looks like the following :
haven't played with Vista MCE yet, but MCE 2005 is definitely lacking when it comes to FM functionality. a 3rd party app could definitely add auto tuning, more presets, fine tuning, and PAR (personal audio recording?) capabilities.
and i'm hoping that newer tuners start adding RBDS (Radio Broadcast Data System ) capabilities. see Designing Video Capture Boards for Use with the Microsoft ActiveX Video Control. this is a way to transmit data over FM. you might have seen how some car stereos or iPods can show the metadata for a currently playing song ... this is done through RBDS. RBDS can also be used to transmit weather and other info which would be useful to have displayed through the MCE interface too. this would be particularly useful in MCE notebooks for when they lack an internet connection, and for the guy that put MCE in his car!
another thing i've seen requested on a newsgroup is being able to listen to the FM tuner while watching TV. e.g. for watching a football game but listening to the announcers on the radio (instead of on TV). this can definitely be done, but i think a dual tuner setup is required. don't think a single tuner can accomplish this, because at least for the TvTuners i have they can only be in FM mode or TV mode. it does not look like they can be in both at the same time. but with dual tuners, you could have the first tuner showing the video only and the second tuner rendering the audio from FM. based on DirectShow's design (haven't actually tried this), you should be able to use both tuners on the same graph and record their output to a single Stream Buffer Engine, to provide live pausing or recording to watch later.
the rest of the article deals with using the TvTuner for TV functionality. NOTE i tested most of this code with both the Hauppauge and Adaptec tuners. even though MCE provides great TV functionality, the first thing i wanted my app to do was TV playback. and with the ability to find an available tuner, the app can play 2 channels at once with the Adaptec tuner (as long as MCE isn't currently using a tuner).
the next step was auto tuning. this is another function i want in MCE. MCE will provide you the guide for all the channels in your area, and then you have to manually edit out the channels that your service doesn't provide. e.g. the guide has all the data for all the premium channels, but i've only got basic cable, so it gave me alot more channels than i really have. so all the app does is loop through all the channels and see if there is a signal present. for analog channels, this actually works really well. for digital channels, i'm not sure if this will work, but the same could be accomplished by grabbing a sample of the channel and doing image processing to see if a channel is present. NOTE the TV channel settings are stored at : C:\Documents and Settings\All Users\Application Data\Microsoft\eHome\EPG\prefs\prefs.xml
to do image processing on the channels, you have to grab a sample (e.g. take a screen capture from the TV channel). this can be accomplished in my graph by replacing the Video Renderer filter with the Video Mixer Renderer 9 filter. it has an interface IVMRWindowlessControl9, which you can call GetCurrentImage on to get a pointer to an image and create a .NET Bitmap out of. once you've got that image you can do image processing to find a valid channel, do skin recognition to find nudity, or even do face recognition. one app that i thought about creating would use an available tuner to loop through each channel and take a recent screen capture, then you could get a view showing a screen capture of many channels at once to choose which channel you wanted to view next. something like this could allow you to quickly find a particular network. what is really interesting about the Video Mixer Renderer 9 filter is that its already being used by MCE. so your app could possibly grab the currently running MCE graph in the ROT, find the VMR9 filter and then grab samples without having to build your own graph. NOTE an alternate way to grab samples is by adding the SampleGrabber interface into a graph. the pic below shows an image sample that has been captured from the video stream
that allows us to grab samples from the tuner, so the next step is to try and push data to the stream. again, this is accomplished with the Video Mixing Renderer 9 filter. you just grab its IVMRMixerBitmap9 interface and call SetAlphaBitmap. my app generates a transparent bitmap with GDI+ that says 'hello world' and then applies it to the currently playing TV channel to show text over the video. you could use this to render your own subtitles. i'm not sure how likely it is for people to create custom subtitles for RecordedTV, but this might make more sense for DVD playback, which uses VMR9 too. the pic below shows overlaying text on the video stream.
finally, i attempted to write the code to change the input used by the TvTuner. both of my tuners support coax, composite, and s-video. DirectShow has the ability to change the input which is used, but i could only get this to partially work. i'm only mentioning this because i've seen a couple newsgroup postings in which people were interested in this functionality.
does any of this to apply to the XBox 360 as an extender? for playback filter graphs, it definitely does not apply. because playback occurs directly on the XBox 360 itself and MCE just transfers the file. so there is no way to hook the actual filter graph on the XBox 360.
but from the FM section, for recorded FM, that recorded file could be played back on an XBox 360.
and from TV, the add-in for showing screen caps of all the channels could work on an XBox 360. the example of hooking VMR9 for overlaying subtitles definitely won't work because that happens at playback. if you wanted to do something like that, then you would have to hook the capture filter graph and insert data at that point.
but that leaves us a handful of ways to use DirectShow to work with the XBox 360. 1) hook the capture graph and add data there. this would be high risk because MCE could easily change the way this works. 2) process an already recorded file to add data. 3) use the tuners to record your own files for playback on the 360.
this article showed some examples of how DirectShow might be used to enhance Media Center. especially for enhancing the FM experience in MCE 2005. and i definitely want to try and implement some of the ideas above in Vista MCE. it's still unknown if Vista MCE will publish its DirectShow filter graphs to the ROT ...
for my first look at DirectShow, i found it somewhat difficult. the hardest part is wading through all the interfaces and filters that exist. it's far from obvious how to know which filters are likely to be on most peoples computers. beyond that, how to handle the differences between hardware and driver implementations. some of their filters have different pins, support different media types, etc... and i spent way too much time trying to get audio to work on the Adaptec tuner. not entirely sure if it's a DirectShow limitation or driver problems, but i hope that DShow ends up providing easier APIs for working with multiple tuners and/or that MCE drivers get tougher certification testing. finally, i want a future version of MCE to allow 3rd party apps to have exclusive access to a tuner if necessary.
the source for the WinForm app is below. remember, i'm a DirectShow beginner, so the code is far from great.
lots and lots of reading. later