Symbolication Tips and Tricks

I’ve probably spent too much of my life symbolicating crash reports. But, it seems that no matter how many times I do it manually, or how many systems I build to automate it, I always end up running into issues. So, I took some time to write down the more interesting tips and tricks I’ve used while working with a symbolication system.

A Quick Intro

Symbolication is the process of mapping an executable address to more human-friendly information. This is typically a function name, but often also includes a corresponding source file and line number. Analyzing a crash report isn’t the only reason to need symbolication, but it’s a very common one. And, an unsymbolicated crash report is much more difficult to use to debug.

Apple has a really excellent article on symbolicating crash reports. It covers the basics, along with a number of other helpful methods of getting and successfully applying symbolic information to a report. I’d recommend checking it out, even if you aren’t manually symbolicating reports. There’s a bunch of helpful stuff in there. And, there’s another, appropriately-named WWDC2021 video that gives a lot more details about how things work at a lower-level, if that’s your kind of thing.

But, this post isn’t an introduction. It’s a look at some of the lesser-known aspects that I wish I had known when I started.

Which Address?

Symbolication is all about looking up the details of an executable address. You start with an address, so how hard could this part be? Unfortunately, there’s actually a little bit of subtlety here that can trip you up. In a crash report, stack traces are gathered for every thread. However, the actual details of how these stack traces are produced matters. Typically, the first (deepest) frame actually represents the current state of the thread. In this case, the address is the instruction currently executing, and is exactly what we want to use for symbolication. The issue is how all the other addresses are captured.

Every frame after the first is captured using a system called stack unwinding. This process is something that typically only debuggers and crash reporters ever need to do. It is surprisingly complex, and believe it or not, actually isn’t possible to do precisely in all situations for all CPU architectures. The addresses produced will usually be address of the instruction after the site of the function call. This makes sense, since upon return of a function, you want to do the next thing, not make the function call again.

Stack unwinding does capture the machine state accurately, it just turns out that this state isn’t exactly what the programmer wants.

Backing up one instruction is easy enough in theory, but for many architectures, instructions are variable-length. This problem is addressed in the DWARF CFI standard, with the suggestion that subtracting one will probably work. At first, this might seem like a totally insufficient heuristic. But, thankfully, our goal is just to get an address in the right region. Even if we are in the middle of one instruction, it’s good enough for symbolication purposes.

Without a fix-up like this, symbolication can produce the wrong line, or even the wrong function entirely, depending on where that address falls. The subtract-one technique is a little unsatisfying, but it works amazingly well, even for densely packed functions written in assembly.

Inline Functions

Inlining isn’t something that comes up too often in normal application development. But, compilers will do it from time to time, and it greatly complicates the symbolication process. When inlining has happened, an address maps to a list of function/file/line groups. While this wasn’t handled particularly well in the past, recent OS/Xcode versions do a great job of correctly symbolicating inlined functions.

4   io.stacksift.InlineTest       	0x00000001027772b4 functionB + 32 (AppDelegate.m:14) [inlined]
5   io.stacksift.InlineTest       	0x00000001027772b4 functionA + 32 (AppDelegate.m:18) [inlined]
6   io.stacksift.InlineTest       	0x00000001027772b4 __45-[AppDelegate applicationDidFinishLaunching:]_block_invoke + 64 (AppDelegate.m:33)

While there was really only one frame in the report, the symbolication process added additional frames to better match the source and programer’s expectations.

Getting this same result using atos is easy, but not only is it not well-documented, it’s very rare to see any examples of the flag that does it. Even Apple doesn’t include it in their examples. You always want to use -i/--inlineFrames. The only reason I can think of that this isn’t the default behavior is tooling based around shelling-out to atos.

Load Addresses

To perform symbolication, you need to determine the load address of the binary containing the executing address. This is given to you directly in a text-based crash report. It’s the first value on this line:

   0x1051f0000 -        0x10528bfff  dyld (852) <1AC76561-4F9A-34B1-BA7C-4516CACEAED7> /usr/lib/dyld

However, as of iOS 14, you could be getting your crash data not in this format, but from MetricKit diagnostics. In that case, there’s a problem that might trip you up. MetricKit data looks like this:

{
    "binaryUUID" : "1AC76561-4F9A-34B1-BA7C-4516CACEAED7",
    "offsetIntoBinaryTextSegment" : 6795280384,
    "sampleCount" : 1,
    "binaryName" : "libdyld.dylib",
    "address" : 6795285912
}

That “offsetIntoBinaryTextSegment”, despite its name, is actually the load address. I’ve filed FB9160176 about this, but I have a feeling we’re stuck with this unfortunately-mislabeled field forever.

Programmatic Symbolication

There are a number of situations where you may want to perform symbolication programmatically. Scripting with atos might be tempting, but there’s a much more powerful way to do it. Apple has a very feature-rich library called CoreSymbolication, and it has been available on macOS for quite a while. Unfortunately, it’s a private framework.

But, don’t let that stop you! It’s much more likely that you’ll need this functionality in an out-of-App Store situation. Stacksift uses CoreSymbolication as part of its symbol-capturing system. We have two open-source libraries based on it. One is a reverse-engineered header + module for the C API. The second is a Swift wrapper, which makes use of that C API. Both are available as SPM modules.

It might feel risky to rely on a private API for this kind of thing, but I’ve been doing it for years now, and the API has proven to be stable and highly-reliable.

Alternatives to the dSYM

It might seem like you need a dSYM to symbolicate because that’s such a common way. But, it actually isn’t required. If you don’t have a dSYM available, you can often use a Mach-O executable directly. This method works with atos and CoreSymbolication, and can be very handy for looking up OS symbols. You’ll only be able to look up function names, but that’s all you need in those cases.

As it turns out, even the dSYM isn’t that great a format for symbolication. It does have the most information, but it also contains a gigantic amount of other stuff, none of which is useful for the symbolication process. Aside from the inefficiency, reverse-engineering an app is much easier when you have a dSYM.

It turns out that the folks that work on LLVM ran into the same problem. They have a defined a format called GSYM that is specifically optimized for symbolication. It isn’t the most well-documented aspect of the project, as far as I can tell, but there’s a lot of interesting info in the original proposal and review. There’s also a fairly full-featured tool available in the project called llvm-gsymutil.

GSYM files can produce the same quality of symbolication as a dSYM, are around 7x smaller, and support a highly-efficient lookup system.

Right now, Stacksift transforms dSYMs before submission into a different, proprietary format. Ours is slightly more space-efficient (around 10x smaller than a dSYM), but has much lower lookup efficiency. It also has a few other minor limitations. And, instead of iterating on that format, we’ve decided to begin the process of migrating to GSYMs. It’s going to be a bit of a journey, but we’ve started with a Go library for parsing and performing lookups against GSYM data.

Wrapping Up

Symbolication is critical for understanding crash reports, but usually developers get to ignore the details. And that’s as it should be. It’s one of those things that you typically only pay attention to when it isn’t working properly. As I’ve worked on symbolication systems, I’ve been in that situation a lot.

Hopefully, some of these tips and details were interesting and maybe even helpful. Let us know if you have any questions or feedback. Even if you sometimes call it desymbolication or symbolification 😉

Stacksift is still accepting beta testers. If you’d like to check out our service, send us an email.

Jul 06, 2021 - Matt Massicotte

Previous: Expanded MetricKit Support
Next: Stacksift is Ready