/cfMDX

comment(s) 

Mobile DirectX with the Compact Framework

http://www.brains-N-brawn.com/cfMDX 5/9/2005 casey chesnut

Introduction

games were my initial draw into the world of computing. i was all about games until discovering that you could download pictures of naked women online :) actually had to give up gaming cold turkey during college, to keep a scholarship that long hours of quake was jeopardizing. never seriously returned to games since then ... but have kept my eyes on them. even loaded up Halo last year and gave it a run. i found it absolutely amazing. wasn't really interested in blasting aliens, but just wanted to run around and play in the world. since then i've been reading a bunch of game-related Artificial Intelligence (AI) books. this is because most AI books on the market are either purely academic or game related. the academic books are just too dry / math heavy for my intent of applying AI to application development. reading the game AI books has only slightly scratched the itch to return to gaming, but it has entirely sparked my interest in using those graphic capabilities for advanced visualization in applications.

the breaking point for me to start learning 3D programming was finally reached when i heard that Managed DirectX (MDX) was coming to the Compact Framework (CF) under the acronym MD3DM (Managed Direct3D Mobile). the next generation of Windows Mobile devices will be able to run CFv2 and MDX apps. i guess this means that we will start seeing PPC devices cobranded with Intel and nVidia/ATI processors. NOTE for development i was able to obtain a device refreshed with a beta of the new OS and a software driver to emulate a graphics processor unit (GPU). The Pocket PC (Windows Mobile 5.0 Beta) SDK also comes with a (next generation) device emulator that can run the samples with software emulation ... but it was pretty slow. graphics emulation in a device emulator ... what else did you expect? so the reality is you must have a device to do MD3DM development.

this article will detail my initial exploration of MDX development with CF. it is my 1st time to play with any 3D graphics API ... so be gentle

Documentation

started out by looking at the WindowsMobile DirectX documentation. it was really thin and just pointed out some of the differences between developing for the desktop and the device. plus it referred to the documentation of the DirectX 9 SDK as being pertinent. so i installed the DirectX 9 SDK for the desktop. the wonderful thing about this installation is that it comes with both the VS.NET flavor of help files as well as CHM files! it actually has 2 CHM files, 1 for native development and a 2nd for managed development. copied both of those over to my Tablet and read the managed flavor. talk about jargon overload. i was lost in a flood of new words. next, i skimmed through the native documentation. still mostly gibberish. then, i read the managed docs again and some of the concepts finally started to click.

then i went through about 5 different Managed DirectX books. the best books are by Tom Miller (creator of Managed DirectX) : Managed DirectX 9 Kick Start and Beginning 3D Game Programming. almost the entire 1st half of the Kick Start applied directly to getting the samples to work below

MSDN Samples

even though the Mobile MDX documentation was lacking, it does come with a good number of samples. the screen shots below show what the samples render

CF only has one assembly for DirectX (Microsoft.WindowsMobile.DirectX). it contains the namespaces

but what happened to DirectDraw, DirectSound, DirectInput, DirectPlay, DirectMusic ... and so on? if you read closely in the DirectX 9 SDK, then you'll see that all of those namespaces have been deprecated. DirectX and Direct3D are about the only survivors

1) CreateMesh is the simplest sample. it just creates a Device and Clears() the screen to the Color.Blue. you can slightly modify that sample to make it render full screen by changing the PresentParameters that are used to create the Device

DisplayMode displayMode = Manager.Adapters.Default.CurrentDisplayMode;
PresentParameters presentParams = new PresentParameters();
presentParams.BackBufferFormat = displayMode.Format;
presentParams.BackBufferWidth = displayMode.Width;
presentParams.BackBufferHeight = displayMode.Height;
presentParams.Windowed = false;
presentParams.SwapEffect = SwapEffect.Discard;

2) Vertices renders the obligatory triangle. it creates a VertexBuffer populated with CustomVertex.TransformedColored vertices. this means it doesnt have to deal with the camera, projection, world views or lighting.

3) Matrices just rotates the triangle along the Y-axis. its VertexBuffer uses CustomVertex.PositionColored vertices. it changes the world view to perform the rotation, but does not deal with lighting.

  

4) Lights creates a VertexBuffer of CustomVertex.PositionNormal vertices representing a cylinder that is rotated by changing the world view. it applies Material to the cylinder and creates both an Ambient and Directional light source

5) Texture changes the vertices to CustomVertex.PositionNormalTextured and applies a banana peel texture to the cylinder. it doesnt use lighting or material

  

6) Meshes has a solution with 2 projects. one of the projects is a MeshConverter console application for the desktop. the MeshConverter will convert a .x mesh file into a .md3dm binary file format. what that means is the .x standard is not supported by Mobile MDX. even worse, neither is the .md3dm format. the sample comes with CF source code (MeshLoader) for reading in a .md3dm file to create a Mesh object, but that MeshLoader class is not a part of the supported SDK. i'm thinking that this pretty much sucks. to add insult to injury, the Mesh class does not provide a Teapot primitive. so you cant just say Mesh.Teapot();. dont ask me ... but it seems that the Teapot is the hello world of 3D programming. regardless, the sample shows how you can load a tiger .md3dm mesh and apply a texture to it.

  

the samples above are the standard Tutorial1 - Tutorial6 that have come with DirectX for some time. the ones below are much more interesting

A) Billboard / BillboardFixedPoint shows a camera moving over terrain. the trees are 2D billboards that are rotated around the Y-axis to face the camera. the texture for the tree is a .DDS texture file. that is interesting because i have had problems getting some other .DDS files to load. i have successfully loaded .BMP, .JPG, .GIF, and .PNG (and some .DDS) images to use as textures in other test projects, but .TIF does not work. NOTE the GraphicsMesh class is useful. i think the BillboardFixedPoint sample is supposed to be faster because reduced floating point math, but its performance was almost identical on the test device i had. the next 2 samples also have FixedPoint samples [CustomVertex.PositionColoredTexture, CustomVertex.PositionTextured ]

 

B) Fractal / FractalFixedPoint generates random terrain with the Fractools class [CustomVertex.PositionNormalColored]

 

C) Lighting / LightingFixedPoint demonstrates a number of different lighting techniques [MyVertex = VertexFormats.Position | VertexFormats.Normal]

 

D) StencilBufferStencilDepth (Device / Emulator / Desktop) demonstrates StencilDepth for determining depth complexity and StencilBuffers. the problem is that it does not work on the test device. it does work on the emulator ... but its too slow. also tried porting it up to the desktop, but it is only rendering the upper-right corner of the screen?

   

E) Text3D shows how to use the Direct3D.Font to render text to the screen. it also has a useful FpsTimerTool class

 

F) Ultimate GMan is a sample side scrolling game that takes user input. needs to have sound added to it [CustomVertex.TransformedColored]

those are just the samples that come with WindowsMobile DirectX. the desktop DirectX 9 SDK has many more samples that you can look at and try to port to MD3DM

Direct3D Tutorials (Pluralsight)

speaking of porting ... thats exactly what i did next. searching for Managed DirectX Tutorials turned up the Direct3D Tutorial Wiki by Craig Andera at Pluralsight. this is a great online tutorial for learning the basics of DirectX. the 16 samples were created for the desktop, but they are pretty simple, so i went through the exercise of porting them to run on WindowsMobile. figured this would be a good way to see some of the differences between the desktop and device implementations. most of the following screen caps will show the device screen cap followed by the desktop screen cap.

1) GameLoop 2) InitGraphics 3) RenderBasics these 1st 3 samples just build up to create the Device and Clear the screen to Black 

started out by creating a SmartDevice PocketPC Windows Mobile 5.0 Device Application. then referenced the Microsoft.WindowsMobile.Direct3D assembly and added the Microsoft.WindowsMobile.DirectX and .Direct3D using statements to the code. but the code would not compile because CF does not have the Form.Created property which is used by the game loop. HACK i got around this by using the Form.Focused property instead. did this because it was the easiest way to get the code to compile, but it is not recommended. the MSDN samples above implement the GameLoop in a different manner that you should use for your own applications. the final step was to change the parameters passes to the Device constructor. the DeviceType only provides the Default enumeration ... easy. the CreateFlags enumeration can be None or MultiThreaded. both of them worked but i stuck with None because i'm new at this. the lines commented out on top are from the desktop version of the code

//using Microsoft.DirectX;
//using Microsoft.DirectX.Direct3D;
using Microsoft.WindowsMobile.DirectX;
using Microsoft.WindowsMobile.DirectX.Direct3D;

//while (app.Created)
while (app.Focused)

//device = new Device(0, DeviceType.Hardware, this, CreateFlags.SoftwareVertexProcessing, pres);
device = new Device(0, DeviceType.Default, this, CreateFlags.None, pres);
//device = new Device(0, DeviceType.Default, this, CreateFlags.MultiThreaded, pres); //works
//device = new Device(0, DeviceType.Default, this, CreateFlags.None | CreateFlags.MultiThreaded, pres); //works

4) CreateVertexBuffer just builds up a VertexBuffer for a triangle. the first change was to remove the VertexFormat declaration because there is no corresponding call for Mobile MDX. the other change was the Pool enum in the constructor for the VertexBuffer. Pool.SystemMemory was the only option that didnt Exception ... so i went with that :) [CustomVertex.TransformedColored]

//device.VertexFormat = CustomVertex.TransformedColored.Format;

VertexBuffer buf = new VertexBuffer(
   typeof(CustomVertex.TransformedColored),
   verts.Length, device, 0,
   CustomVertex.TransformedColored.Format,
   //Pool.Default - original
   //Pool.Managed - exceptioned on test device
   //Pool.VideoMemory - hardware only?
   Pool.SystemMemory);

5) RenderVertexBuffer renders the VertexBuffer created above to the screen. didnt have to make any additional changes. er, um ... i dont think this sample could be made any more gay. a rainbow colored triangle on a Bisque background? [CustomVertex.TransformedColored]

6) CoordSystems just explains how Matrix math can be used to change the projection, camera, and world views [CustomVertex.TransformedColored]

7) Render3D renders the rotating triangle to the screen by setting the views. the triangle is black because there is no lighting [CustomVertex.PositionColored]

8) Lighting adds ambient and directional lighting. the only code change was on the directional light. this change was already pointed out by the article and has to be done for the latest releases of the DirectX SDK on the desktop too [CustomVertex.PositionNormalColored]

//device.Lights[0].Commit();
device.Lights[0].Update();

9) Texture renders a bitmap to the triangle. the only change was for how the texture loads from disk. i cant believe we still have to do the Assembly.GetExecutingAssembly().GetName().CodeBase nonsense in CFv2 [CustomVertex.TransformedTextured]

//Texture t = TextureLoader.FromFile(device, "texture.bmp");
string codeBase = System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase;
string currentDirectory = System.IO.Path.GetDirectoryName(codeBase);
Texture t = TextureLoader.FromFile(device, currentDirectory + @"\texture.bmp");

10) Materials creates a cylinder with material and lighting. the only change was due to the CustomVertex class not having the SetPosition and SetNormal helper methods [CustomVertex.PositionNormal]

//verts[2 * i].SetPosition(new Vector3((float)Math.Sin(theta),-1,(float)Math.Cos(theta)));
SetVertexPosition(verts, 2 * i, (float)Math.Sin(theta), -1, (float)Math.Cos(theta));
//verts[2 * i].SetNormal(new Vector3((float)Math.Sin(theta),0,(float)Math.Cos(theta)));
SetVertexNormal(verts, 2 * i, (float)Math.Sin(theta), 0, (float)Math.Cos(theta));
//verts[2 * i + 1].SetPosition(new Vector3((float)Math.Sin(theta),1,(float)Math.Cos(theta)));
SetVertexPosition(verts, 2 * i + 1, (float)Math.Sin(theta), 1, (float)Math.Cos(theta));
//verts[2 * i + 1].SetNormal(new Vector3((float)Math.Sin(theta),0,(float)Math.Cos(theta)));
SetVertexNormal(verts, 2 * i + 1, (float)Math.Sin(theta), 0, (float)Math.Cos(theta));

protected void SetVertexPosition(CustomVertex.PositionNormal[] verts, int index, float X, float Y, float Z)
{
	verts[index].X = X;
	verts[index].Y = Y;
	verts[index].Z = Z;
}

protected void SetVertexNormal(CustomVertex.PositionNormal[] verts, int index, float Nx, float Ny, float Nz)
{
	verts[index].Nx = Nx;
	verts[index].Ny = Ny;
	verts[index].Nz = Nz;
}

11) ZBuffers for depth. this didn't require any code changes [CustomVertex.PositionNormalColored]

12) Mesh was supposed to render a tank mesh. i didnt happen to have a tank mesh (and corresponding texture) on me, so i used the tiger mesh and .md3dm texture from the MSDN mesh sample. the big change was to handle the loading of the mesh and associating of its texture in CreateMesh(). this involved adding the MeshLoader class to the project to read the .md3dm file. finally, i had to change the way Color for Ambient light was declared to get the tiger to not render as all black (panther?). NOTE instead of a teapot ... a tank should be a primitive so i can just do Mesh.Tank(); in my code. which calls for a classic bill and ted flashback before the code

[ falling down a really big hole ]
Dead Bill : Hey Ted? Wanna play 20 questions?
Dead Ted : Okay! I got one!
Dead Bill : Is it a mineral?
Dead Ted : Yeah!
Dead Bill : Are you a tank?
Dead Ted : Whoa! Yeah!

//CreateMesh(@"tank.x");
CreateMesh(@"Tiger.md3dm");

protected void CreateMesh(string path)
{
	//ExtendedMaterial[] exMaterials;
	//mesh = Mesh.FromFile(path, MeshFlags.SystemMemory, device, out exMaterials);

	// Load the mesh from the specified file
	string[] textureFilenames = null;
	string meshPath = "Mesh." + path;
	Stream meshStream = Assembly.GetExecutingAssembly().GetManifestResourceStream(meshPath);
	mesh = MeshLoader.LoadMesh(device, meshStream, MeshFlags.SystemMemory, out materials, out textureFilenames);
	meshStream.Close();

	if (textures != null)
	{
		DisposeTextures();
	}

	/*
	textures = new Texture[exMaterials.Length];
	materials = new Material[exMaterials.Length];
	for (int i = 0; i < exMaterials.Length; ++i)
	{
		if (exMaterials[i].TextureFilename != null)
		{
			string texturePath = Path.Combine(Path.GetDirectoryName(path), exMaterials[i].TextureFilename);
			textures[i] = TextureLoader.FromFile(device, texturePath);
		}
		materials[i] = exMaterials[i].Material3D;
		materials[i].Ambient = materials[i].Diffuse;
	}
	*/

	// Extract the material properties and texture names
	textures = new Texture[materials.Length];
	for (int i = 0; i < materials.Length; i++)
	{
		// Set the ambient color for the material (D3DX does not do this)
		materials[i].Ambient = materials[i].Diffuse;
		// Create the texture
		string texturePath = "Mesh." + textureFilenames[i];
		Stream textureStream = Assembly.GetExecutingAssembly().GetManifestResourceStream(texturePath);
		textures[i] = TextureLoader.FromStream(device, textureStream);
		textureStream.Close();
	}
}

//device.RenderState.Ambient = Color.FromArgb(0x00, 0x00, 0x00);
device.RenderState.Ambient = Color.White;

13) DeviceRecovery demonstrates how the app can recover from the Device being lost. similar code can also be found in the Mobile MSDN samples. i tested this code out on the device and it seemed to work correctly when opening other applications and then returning to the DirectX sample. the problem is that it did not work when switching to Landscape mode. actually, i could not get any sample to run in Landscape mode at all. it might be harder to use the controls in landscape mode, but i hope that it will ultimately be supported. NOTE MD3DM cannot support Landscape mode because the Native D3DM implementation does not support landscape either. So your application should be aware of the orientation on startup and capture when the user tries to change orientation. NOTE both screen caps below are from the device emulator to show how landscape fails [CustomVertex.PositionColored]

14) IndexBuffers shows a rotating multi-colored cube. this sample worked fine on the desktop but was rendering all black on the device. the change i had to make involved the projection view [CustomVertex.PositionColored]

//device.Transform.Projection =
//  Matrix.PerspectiveFovLH((float)Math.PI / 50.0F,
//	this.Width / this.Height, 1.0F, 100.0F);
device.Transform.Projection =
  Matrix.PerspectiveFovLH((float)Math.PI / 50.0F,
  1.0F, 1.0F, 100.0F);

15) CreateMesh shows how to build up a cube mesh in code. there were a number of code changes to get this to work, including grabbing some of the code from the MeshLoader class. the first problem involved the creation of the Mesh object. had to change both MeshFlags and VertexFormats. MeshFlag.SystemMemory threw an exception even though it is supposed to equivalent to the OR operation below. VertexFormat.Position and .Texture0 both exceptioned when creating the mesh. tried the other overloads but they did not render a cube. would end up with a pyramid most of the time. these problems are related, but the next step was to get back not having the mesh.SetVertexBufferData and mesh.SetIndexBufferData methods. tried to use mesh.VertexBuffer.SetData and mesh.IndexBuffer.SetData instead, but they rendered incorrectly with the vertexFormat set above. so i commented those out and had to do it manually. ended up setting the vertexFormat to Texture1 and just didnt set the texture values ... lame. then i wrote my own SetBufferData() method. this was a big mess ... and i think the problem is in the Mesh constructor exceptioning on many of the vertexFormats [CustomVertex.PositionOnly]

//mesh = new Mesh(
//  12,                       // 12 faces
//  8,                        // 8 vertices
//  0,                        // no flags
//  VertexFormats.Position,   // Position information only
//  device);   		
mesh = new Mesh(
  12,                       // 12 faces
  8,                        // 8 vertices
  MeshFlags.VbSystemMem | MeshFlags.IbSystemMem,
  flags,
  VertexFormats.Texture1,   
  device);

//mesh.SetVertexBufferData(vertices, LockFlags.None);
//mesh.SetIndexBufferData(indices, LockFlags.None);
SetBufferData(mesh, vertices, indices);
//mesh.VertexBuffer.SetData(vertices, 0, LockFlags.None); //rendered wrong
//mesh.IndexBuffer.SetData(indices, 0, LockFlags.None); //rendered wrong

//load vertices and indices into mesh
public void SetBufferData(Mesh mesh, CustomVertex.PositionOnly[] vertices, short[] indices)
{
	// create the index and vertex buffer
	VertexBuffer vb = mesh.VertexBuffer;
	IndexBuffer ib = mesh.IndexBuffer;

	int bytesVertices = VertexInformation.GetFormatSize(VertexFormats.Texture1) * vertices.Length;

	byte[] rgbVb = GetVertexBytes(vertices, bytesVertices);
	// write the vertex data
	GraphicsStream gsVb = vb.Lock(0, bytesVertices, LockFlags.None);
	gsVb.Write(rgbVb, 0, bytesVertices);
	vb.Unlock();

	byte[] rgbIb = GetIndexBytes(indices);
	// write the index data
	GraphicsStream gsIb = ib.Lock(0, rgbIb.Length, LockFlags.None);
	gsIb.Write(rgbIb, 0, rgbIb.Length);
	ib.Unlock();

	mesh.LockAttributeBuffer(LockFlags.None);
}

16) Fonts rotates some text around on the screen.

this displays some 2D text and 3D rotating text. the 2D text part did not require any changes, but i dont think Mobile MDX can support 3D text at all because it doesnt have the method to create a mesh for a font. i'm really ok with that ... because rotating text is not cool. the only code changes involved commenting out the 3D text parts

 

A) SaveToFile is for saving a screen cap of what the device is rendering. the 1st problem is that i could not get Mobile MDX to bind to a Panel or PictureBox (error 80004005). i can only get it to bind to a the entire Form. NOTE MD3DM can only bind to the top level Form. after that, i was able to grab the BackBuffer for the device, but could not determine how to access the bits so that i could save them to an image? davidwr ended up walking me through the code below to be able to save a screenshot

Surface backBuffer = _device.GetBackBuffer(0, BackBufferType.Mono);
Rectangle rect = new Rectangle(_device.Viewport.X, _device.Viewport.Y,
   _device.Viewport.Width, _device.Viewport.Height);
Surface imageSurface = _device.CreateImageSurface(_device.Viewport.Width, _device.Viewport.Height, backBuffer.Description.Format);
Point destinationPoint = new Point(0, 0); //TODO?
_device.CopyRects(backBuffer, rect, imageSurface, destinationPoint);
int pitch;
GraphicsStream gs = imageSurface.LockRectangle(rect, LockFlags.ReadOnly, out pitch);
Bitmap b = new Bitmap(_device.Viewport.Width, _device.Viewport.Height, PixelFormat.Format16bppRgb565);
BitmapData bd = b.LockBits(new Rectangle(0, 0, b.Width, b.Height), ImageLockMode.WriteOnly, PixelFormat.Format16bppRgb565);
byte[] baGs = new byte[gs.Length];
gs.Read(baGs, 0, baGs.Length);
IntPtr ptr = bd.Scan0;
int bytes = b.Width * b.Height * 2; // 3;
System.Runtime.InteropServices.Marshal.Copy(baGs, 0, ptr, bytes);        
b.UnlockBits(bd);
b.Save(dialog.FileName, ImageFormat.Bmp);
imageSurface.UnlockRectangle();

Smartphone

all of the projects / samples so far were dealing with Pocket PCs, but the next generation of Smartphones will support MD3DM too. to test this out i created some SP projects. they worked without any changes except TextureLoader seemed to be overly case sensitive, which did not occur in the original PPC project. the pics below show 3 of the converted projects running in the SP emulator.

AdapterCaps

went ahead and created a simple little app to grab the DeviceCaps for Mobile MDX. ran it both on the device (left), SP emulator (middle) and PPC emulator (right). a text file of the full results is linked below. if it is unsupported, then the capability is preceded by a (-) symbol. you can see that the emulator supports alot more functionality than the test device does.

device.txt | PPC_emulator.txt | SP_emulator.txt | desktop.txt

Conclusion

MD3DM has a promising start. it ended up being pretty trivial to port samples intended for the desktop over to the device. so the power of the API doesnt look like it will be in question. the question that remains is how it will perform on the next generation of devices. MDX on the desktop has been entirely beating my performance expectations, so i hope that Mobile MDX will do the same. guess i just have to wait until i get one of the new devices in hand to know for sure

Source

this is the ported code from the Pluralsight Direct3D Tutorial. the solution is from a slightly early release of VS 2005 B2

Updates

none planned

Future

this article was finished weeks before MEDC 2005, but i held off releasing it until the public announcements. the follow up article will be out within a week. plus i've got a ton of ideas for other technologies ... especially with the drop of B2