My solution to those problems has been for a while to set a common build directory for my entire user. This preference is set in the Xcode preferences. Having a single build directory has quite a number of advantages (one-shot delete of temporary build files, Spotlight ignore for faster build times, one directory to go to for Terminal sessions, one directory to add to dyld environment variables, etc). It also happens to fix all sub-project related issues, because every project suddenly has the same build directory.

However, a project should not assume that this will be the case. Most external contributors will not have set that Xcode preferences, or they may have a good reason for not doing so. In any case, a project should always build successfully no matter the specific environment, given a sane Xcode tools installation.

So let’s look at some of the more annoying issues, and how we can resolve them in a manner that works in both situations (common build directory or not).

Framework header files

Despite the sub-project properly added to the master project, and even though linking phases will properly resolve the path of any shared library or framework reference from an sub-project, including framework header files will fail. This comes as a bit of surprise, because adding a framework as a regular project item, no matter where it is on disk, will allow framework includes to work for that particular framework.

The solution to this problem is of course to specify the sub-project’s build directory in the framework search path. An additional twist is to use Xcode build setting variables to handle all configurations at once. One does need to assume certain things about the sub-project’s location and build settings, but for the common case of having the sub-project in a sub-directory below the “master” project, my solution will do the job.

The generic pattern is something like this:

FRAMEWORK_SEARCH_PATHS = "$(SRCROOT)/SUB-PROJECT-DIRECTORY-NAME/build/$(CONFIGURATION)";

where SUB-PROJECT-DIRECTORY-NAME is the name of the sub-directory containing the sub-project. As an example, Riven X’s Base.xcconfig contains the following:

FRAMEWORK_SEARCH_PATHS = "$(SRCROOT)/mhk/build/$(CONFIGURATION)";

Although I have no experience with shared library include files, I also strongly suspect including them will fail. Their case is a bit different, considering the header files will not be bundled, but the gist of the problem remains the same. Adapting my solution for frameworks to shared library is a (trivial) exercise for the reader.

Copying sub-project build products

For some unknown reason (and most likely not a very good one), using a sub-project build product reference in a copy phase in the “master” project will fail. The Xcode build system construct an incorrect path, assuming the referenced build product is in the “master” project’s build directory.

The solution is to use a shell script to copy that build product. As an additional requirement, I wanted to use Xcode’s copy tool (pbxcp) in exactly the same way as the build system does. The script makes the same assumption as the solution for framework header files.

First, the code. I’m not going to make it generic, so this is for copying MHKKit to the Riven X bundle. You can of course copy more than one item per script invocation with proper modifications. On with the code:

# Find the pbxcp tool
COPY_TOOL="${DEVELOPER_LIBRARY_DIR}/PrivateFrameworks/DevToolsCore.framework/Resources/pbxcp"
if [ ! -f "${COPY_TOOL}" ]; then
	COPY_TOOL=”/System/Library/PrivateFrameworks/DevToolsCore.framework/Resources/pbxcp”
fi

# Find the build product
MHKKIT_PATH=”${BUILT_PRODUCTS_DIR}/MHKKit.framework”
if [ ! -d "${MHKKIT_PATH}" ]; then
	# assume default Xcode setup where MHKKit was built in its own build directory
	MHKKIT_PATH=”${SRCROOT}/mhk/build/${CONFIGURATION}/MHKKit.framework”
fi

# Copy the build product
$COPY_TOOL -exclude .DS_Store -exclude CVS -exclude .svn -resolve-src-symlinks “${MHKKIT_PATH}” “${BUILT_PRODUCTS_DIR}/${FRAMEWORKS_FOLDER_PATH}”

The script first finds pbxcp (in a way that will work on both 10.4 with Xcode 2.4 and above, and 10.5 with Xcode 2.5 and 3.0). It then assumes the build product (in this case, “MHKKit.framework”) is in the “master” project build directory. If that turns out not to be the case, it assumes it is in the default build directory of the sub-project (in this case, the MHKKit project) for the active configuration. It finally invokes pbxcp to copy the build product to the appropriate location (in this case, the “Frameworks” directory of the Riven X application bundle).

For a list of build setting variables for common bundle locations, see Xcode Build Setting Reference, specifically the “Product Layout Build Settings” category.

As noted by Michael Tsai, NSNotificationCenter is a singleton, and thus essentially is one big honkin’ global. Globals are bad ™.

This was posted by Andy Finnell echoing Michael Tsai.

But in fact, NSNotificationCenter isn’t a singleton. You can create as many as you want. I agree that sending every notification to the default center isn’t a good idea. It turns it into a monolithic entity it isn’t meant to be. The default notification center should be kept only for communication with the rest of Foundation and AppKit.

However, if you are careful in your designs, you can easily have sub-system notification centers which only certain class clusters know about. That way, you get the benefits of NSNotificationCenter without the problems that have been underlined by several people already.

That being said, there are a few annoyances with using NSError, one of which is constantly having to check if the NSError ** variable your method was given isn’t nil before you dereference it to assign it a new NSError object. There’s also the verbosity of creating those instances.

So in order to make my life a little bit easier, I wrote a set of macros that cover the specific case where you want to create an NSError instance and then bail out of the method. They check if the NSError ** is nil or not, create an NSError instance and assign it to the reference (or set the reference to nil for the no error ones), then return a provided value to the caller. It’s important to note that you shouldn’t use a semi-colon after them, since the macros expand to a block ({…}). This is necessary for the macros to work correctly in situations such as if statements.

You can download the header file from the MPQKit Trac wiki. There are no license restrictions whatsoever.

I originally anticipated to do that stuff in PHP, but I figured it would be nice to learn something new, so I finally settled on Django, the “Web framework for perfectionists with deadlines”. Django is written in Python, which is a really nice language, not to mention hip at geek parties.

In any case, it hit me last weekend I would have to load World of Warcraft saved variables files, which are just Lua files declaring variables in the global scope. I thought this would be an ordeal and didn’t look forward to writing parsing code, or worse, a formal grammar.

Turns out Lua is so similar to Python a few regular expressions did the trick. I have to admit, I was rather pleased by the simplicity of the solution

So there you have it: the wowsv Python module. It exports a single function, aptly named wowsv, to load World of Warcraft saved variables into Python. You can do whatever you want with it, I don’t really care. Just credit me if you’re going to use it, and of course I’d appreciate to know about any changes or improvements you might make to it.

Download wowsv (sha1: 4711ce5a69e774b973bc5de2975051e18da11ee5)

In case you were wondering about the compression, my server would try to execute a straight .py as a CGI script.

Why depth testing doesn’t cure cancer

Depth testing can be extremely useful in 3D applications, but Riven X is only a simple 2D compositing engine. As such, depth testing isn’t required to obtain the desired image but only as an optimization technique. So for depth testing to make sense, it needs to offer a net advantage over other methods, which is far from being obvious.

First of all, depth testing has a memory cost. Not only must you maintain a set of z coordinates for all your primitives, you must also instruct OpenGL to maintain a depth buffer as part of its framebuffer. It is true that statistically there will never be a large number of primitives in a given card, however it is also likely Riven X will need to send large textures to the GPU each frame (such as for movies). Consequently, memory bandwidth to the GPU becomes a precious commodity that needs to be managed tightly.

Secondly, depth testing adds a level of complexity that other more naïve solutions do not have. Specifically, because Riven X is a 2D composite, there is no depth information in a card’s description. As such, it would be necessary to generate z coordinates for each renderable element before depth testing could be used to control display order. This would essentially boil down to a simple quantization problem (m elements to render, n possible depth values). Nothing to worry about but why do more when you can do less? In addition, elements can be enabled and disabled. That implies computing a new depth value whenever an element is enabled, and eventually a complete re-quantization of all enabled elements when you reach the upper bound of the dynamic range of your depth values.

Finally, depth testing doesn’t solve the what to render problem entirely. Indeed, depth testing can be used to filter out “too far” objects, but you need to define your “far plane” or “depth clipping value” and initially set the depth value of all your primitives to that cutoff threshold.

The texture problem

If you recall from the previous entry, one of the big advantages of depth testing is the ability to render a large number of primitives in one OpenGL operation (using vertex arrays and the likes of glDrawArray) that share common state properties. This is the typical case of 3D applications, where models are aggregates of various types of primitives (triangles, strips, quads, etc). However, this is not applicable to Riven X because each primitive is essentially used as a billboard on which a unique texture can be applied. Typically, a 3D model will be made of a few sub-sections, each of which is made of n primitives and one texture (or a few with multitexture) that will be applied to all the primitives using each primitive’s texture coordinates. If I were to do the same in Riven X, I’d have to dynamically generate a large composite texture with each element’s texture in it. Because each of those sub-regions will be rectangular, it is likely I’d run out of texture space rapidly. And that doesn’t include the fact I’d have to write code to do the composite in an optimal fashion. Thermodynamics win, solution dismissed, I remain in my lazy state.

A simple array

Riven X uses a simple enabled elements array to drive the card rendering loop, which looks like this:

NSEnumerator *renderListEnumerator = [_renderList objectEnumerator];
id anObject = nil;
Class lastRenderableClass = nil;

// bind the picture texture and draw the quad
glBindTexture(GL_TEXTURE_RECTANGLE_EXT, _pictureTextureObjects[pictureIndex]);
glDrawArrays(GL_QUADS, pictureIndex, 4);
}
}

It uses the class of each element to know how to render it (right now, I’m only rendering pictures) and tries to minimize state changes as much as possible. An array is a relatively simple solution that doesn’t offer fast deactivation but relatively OK activation. It should be noted that deactivation wouldn’t be fast either if I was using depth testing (both are linear), which may be fixed in both instances using a reference table from element resource ID to element index.

However, sampling indicates Riven X renders a lot more often than it enables or disables elements, so this isn’t a significant problem.

So this is how it works for now, and until I have a more complete application that I can sample in a more realistic manner, an array will do just fine.

Although I don’t believe in forcing warnings into errors (some warnings can legitimately be ignored), I do make it a point to kill as many warnings (if not all of them) in code I ship (milestones, releases). Obviously, warnings will be abundant during development, but it should be just as important to get them solved as it is to debug and sample before pushing a program to the public.

If one reads the Riven X project page, one can read “but I expect something barely playable in November”. Obviously, that’s not going to happen. Without going into boring details, let’s just say it’s been one hell of a semester.

So I thought I’d discuss a particular issue that’s been annoying me with Riven X: how to render a card. Specifically, how to efficiently render what needs to be rendered in a way that will yield the proper display order while allowing reasonably fast object rendering activation and deactivation.

Background theory

Essentially, Riven is made of stacks and cards. Just like in real-life, a stack contains multiple cards. But what does a card contain? Essentially, renderable objects: pictures, movies, sound effects. Obviously, sound effects are completely different from pictures and movies because they are strictly acoustic in nature. That leaves us to deal with pictures and movies as far as graphics are concerned. The biggest problem by far is to respect the display order. You see, initially a card is blank: it has no active renderable objects. But as scripts are executed in reaction to game state changes or user events, renderable objects start getting activated and deactivated. The rule of the game is then simple: whatever gets activated last is drawn on top of everything. In other words, you “pile things up” as you activate them. Deactivating an item means removing it from the stack while preserving the order of the other elements. We can make a simple diagram to illustrate this:

This seemingly simple problem has caused me more than one headache, and I’m still unsure the solution I settled upon will be satisfactory. Only sampling will tell, and I’m far from being able to do that. At this point, you may continue reading for details about potential solutions, or wait for my next entry which should be concerned with the solution I have chosen and the unavoidable reasons supporting that choice.

How are card objects stored?

This may not be final, but the current organization of data in cards is presently as follows:

1. A single vertex buffer object (VBO) for all the picture coordinates. This includes 4 vertex coordinates (one for every vertex defining the region of a picture, or the quad in OpenGL parlance) and 4 texture coordinates (essentially, a texture coordinate is a 2 component vector (for 2D textures anyways) that maps a particular point in a texture (in normalized coordinates – -1 to 1) to a particular vertex) for each picture.
2. An array of texture objects, one for each picture. The order of this array matches the order in the coordinates VBO.
3. An array of GLMovie objects, each of them using one VBO for vertex coordinates, one VBO for texture coordinates and one texture object.

How does storage affect rendering?

There are quite a large number of factors we could enumerate here, but the principal one has to do with minimizing state changes. OpenGL is essentially a large state machine (this is somewhat less true with the OpenGL Shading Language, but we’ll overlook that), and everytime you change the state, there is an associated cost. Some operations are more costly than others, but the general rule is to render such that state changes are minimized. Of course, one must also get accurate rendering, so not only must you respect the former rule, you need to do so in a way that preserves display order.

With that in mind, it should be relatively clear that in the case of a card, we’d like to bind (make current, activate, select) the picture VBO, render all enabled pictures (one at a time unfortunately, since we need to bind the correct texture object for each picture), then render all the movies. The next logical step is therefore to analyze how OpenGL renders a scene.

From commands to images

Without anything fancy added to it, OpenGL will display or draw primitives as they are submitted to it. In other words, OpenGL’s display order is OpenGL’s render order. However, to minimize state changes, it is often necessary to change the render order to group primitives that share some amount of state information. For example, if you have a red quad below a green quad below a second red quad, you’d normally render the first red quad first, then the green quad and finally the second red quad. However, it would be more efficient if you could render both red quads together, then change the front color and render the green quad. Unfortunately, if you do that, you’ll mess up your scene. OpenGL’s solution to this problem is called the depth buffer.

Give me depth

The depth buffer is essentially a two dimensional grayscale texture the same size as the OpenGL viewport (the area of the screen or window OpenGL is rendering into), which means there is one pixel in the depth buffer for every pixel in the viewport. Normally, a value of 0 means closest to the viewer and a value of 1 means as far away from the viewer as possible. Usually, the depth buffer is cleared with 1 at the beginning of every frame, in the same fashion as the color buffer.

With the depth buffer made writable and depth testing enabled, you can start specify a depth coordinate (also known as a z coordinate) for your primitives. That depth coordinate, once it is properly modified by applicable transform matrices and extrapolated for every fragment generated by rasterizing a given primitive, will be compared to the corresponding value already found in the depth buffer. If the new depth value passes the depth test, which usually means if it is closer to the viewer than the previous value, the fragment continues to be processed. Otherwise, the fragment is altogether dismissed, with the final effect of preserving whatever was already there in the final image that will be displayed on your screen.

In short, depth testing allows us to control display order while optimizing render order! It seems we have found our solution, but unfortunately things are not quite so straightforward. As will be made clear in an upcoming entry, depth testing has problems of its own. Stay tuned!

We have reports from Brent Simmons and renowned drunkenbatman (and he’s got a pretty good one too). More bugs from Chris Woods, Mike Zornek. I’ll try to add more links to this entry as bugs are posted, but it’s a bit hard to track them at the moment (someone should make a website to track those).

In any case, I was supposed to write about a QTKit bug or limitation having to do with opening movies embedded inside files at a particular known offset. After several hours of work, I can say with relative confidence that QuickTime is one awesomely weird piece of technology. For reasons that are beyond my understanding, trying to open a perfectly valid movie file using a data fork reference number and NewMovieFromProperties fails with a movie not found error (-2048). Creating a movie for the same file using a POSIX path or a CFURL works perfectly fine. Of course, since QTKit doesn’t expose functionality equivalent to kQTDataLocationPropertyID_DataFork, there is no such equivalent problem in QTKit.

Which brings me to the code I posted in On Report-an-Apple-Bug Fridays and to my current state of confusion. The snippet I posted is part of MHKKit, a Foundation-based framework I am currently developing as part of the Riven X project, a Mac OS X native re-implementation of the popular game by Cyan Worlds. Riven data files contain several embedded QuickTime movies, and the only API that properly opens them is NewMovieFromProperties with kQTDataLocationPropertyID_DataFork and kQTMovieResourceLocatorPropertyID_FileOffset (which, incidentally, only works with kQTDataLocationPropertyID_DataFork). Pretty cute, isn’t it? But I get the message Apple, I won’t try to see logic in QuickTime anymore. I promise.

I still have a bug to post

But I still have a bug to post, although I’m almost certain it will finish as a duplicate because it’s such an obvious one. It involves the Dock and Uniform Type Identifiers. The theory is that UTIs are as of Tiger the primary way to determine the type of a items on the system, be them files, filesystem objects or more abstract entities like contacts or calendar events. Everything else (OSTypes, extensions and MIME types) maps back to a UTI. Therefore, if one is write a Tiger-only application, one should normally use UTIs for purposes of file type identification. If you’re not familiar with UTIs, I encourage you to read the link above which explains in good details what they are about. The executive summary is that they are a most worthy successor to every other type identification mechanism we’ve had so far.

One area in particular where UTIs should be used is in the Info.plist file that can be found in every bundle when it’s time to declare the types of documents or entities a particular application supports. This is the well-known CFBundleDocumentTypes key in plists. Instead of using CFBundleTypeExtensions or CFBundleTypeOSTypes, one should use LSItemContentTypes, which is specified as an array of UTIs corresponding to the document type entry it belongs to(CFBundleDocumentTypes is an array itself). Unfortunately, this isn’t quite usable yet because the Dock doesn’t perform conformance tests properly.

If you read the documentation about UTIs, you know they are organized in hierarchies. This implies some form of relationships between UTIs, which is called conformance. A UTI conforms to another UTI if it expresses a refinement of the conformed to type. A very simple example of conformance is with JPEG images. It’s readily apparent that a JPEG image is generally speaking an image, and thus the public.jpeg UTI conforms to the public.image UTI.

Conformance is really important and amazing because for example an image editing application doesn’t need to declare a document type for every known image format – all it needs to do is declare one document type with the UTI public.image and suddenly the system knows that particular application can open any image file.

We call a conformance test the process of computing whether a particular UTI conforms to another given UTI. For instance, a conformance test of public.jpeg and public.image would indicate public.jpegconforms to public.image. Similarly, a conformance test of public.mp3 and public.image would fail.

You can now most likely understand the bug when I say that the Dock doesn’t perform conformance tests properly. What happens is that if your application declares a document type with a UTI, say public.jpeg, the Dock will accept a file drop on that application’s icon only if the UTI of the file being dropped is exactly public.jpeg. If your document type uses public.image, the Dock will refuse the file drop. On the other hand, this works perfectly fine in the Finder.

Radar bug #4234661.

If you reported a bug to Apple today—it being Report-an-Apple-Bug Friday—please leave a comment and say what your bug id is. (Oh, and why not say what the bug is too. But we gotta have an id—a bug id is a badge of honor.)

I think weekly bug reports are a pretty good idea and so I will try to follow this new tradition among Mac developers as dutifully as I can. My first one will probably be about either a limitation or a bug in QTKit related to movie creation.

The problem has to do with opening embedded QuickTime movies – that is, opening movies that are somewhere inside another file. This is supported by QuickTime through various means:

• You can either open the movie using an existing fork, or
• you can open the movie using a file reference of some sort

In both cases, you specify the offset of the QuickTime movie when the movie is created, which in our case we assume is a known correct value. The best way to open a movie in such a way with QuickTime 7 is by using the new NewMovieFromProperties function. The following code demonstrate how this can be done. It’s an excerpt from MHKKit, a framework project I will soon introduce in more details.

// setup the movie properties
Boolean active = TRUE;
QTVisualContextRef visualContext = NULL;
QTNewMoviePropertyElement newMovieProperties[] = {
{kQTPropertyClass_DataLocation, kQTDataLocationPropertyID_DataFork, sizeof(forkRef), &forkRef, 0},
{kQTPropertyClass_MovieResourceLocator, kQTMovieResourceLocatorPropertyID_FileOffset, sizeof(qt_offset), &qt_offset, 0},
{kQTPropertyClass_Context, kQTContextPropertyID_VisualContext, sizeof(visualContext), &visualContext, 0},
{kQTPropertyClass_NewMovieProperty, kQTNewMoviePropertyID_Active, sizeof(active), &active, 0}
};

// make the movie
Movie aMovie = NULL;
err = NewMovieFromProperties(sizeof(newMovieProperties) / sizeof(newMovieProperties[0]), newMovieProperties, 0, NULL, &aMovie);

In this particular case, I use an existing fork to open the movie, but using a URL or FSRef to the file from which that fork was opened shouldn’t make any difference in theory. That is unfortunately where QTMovie seems to be broken. Attempting to create an instance of that class with - initWithAttributes:error: and the QTMovieFileOffsetAttribute attribute fails, even if the specified file and offset are the same.

I haven’t done any digging in order to figure out exactly what is broken, but I intent to before posting my bug report. Remember kids: detailed bug reports make QA people happy and bug squishing a joy.

Since I’m insane (more than most, in any case), I wrote a bash shell script. Here it is:

#!/bin/bash
##
## svn-backup.sh -- Daily backup script for subversion repositories
## Copyright (c) 2005, Jean-Francois Roy
##

echo “svn-backup.sh completed at `date`”
echo “”