Get crash logs on mobile device

18 Aug 2016

Every iOS developer knows that we can get crash information while running application via Xcode, including simulator and mobile device. Basically this would help us to debug under most conditions, however sometimes crashes are not discovered by us, like testers. Thus, we need to fetch these crash logs from their devices.

There will be several steps to get crash log.

First of all, connect the target device to your Mac, sync via iTunes.

Then, open directory ~/Library/Logs/CrashReporter/MobileDevice/YourMobileName/Retired, and you will see files named like BundleName-Time.crash. These files are generated by mobile when the application crashed, this would help us to find out what’s going on.

Quite easy, isn’t it? Well, let’s see what’s in it. A crash log would looks like this(a crash report demo):

Incident Identifier: 80F61CF6-CECC-4E9D-A85E-03F68BBBBC73
CrashReporter Key:   383daadfbd4ac97cf2e53531fc349858fb09ae3c
Hardware Model:      iPhone6,2
Process:             MyAppName [6485]
Path:                /private/var/containers/Bundle/Application/9DB38223-0B3F-4F9E-AAFA-BD025B7C2602/MyAppName.app/MyAppName
Identifier:          com.myCompany.MyAppName
Version:             1 (5.2.12)
Code Type:           ARM-64 (Native)
Parent Process:      launchd [1]

Date/Time:           2016-06-14 14:23:25.25 +0800
Launch Time:         2016-06-14 14:23:23.23 +0800
OS Version:          iOS 9.3.2 (13F69)
Report Version:      105

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  0

Filtered syslog:
None found

Last Exception Backtrace:
(0x181c9edb0 0x181303f80 0x181c9ec80 0x182624154 0x1005461f8 0x181ca4a60 0x181b9c488 0x1857ba294 0x1857b9e38 0x1857bb030 0x185362ae8 0x1857a1d38 0x18579bed8 0x1856c79fc 0x18536683c 0x1853665ec 0x185756658 0x100556ff0 0x181ca2a1c 0x181ba0c5c 0x186dfcc40 0x186dfc844 0x100107044 0x1872d3a50 0x187540b84 0x1002c50e4 0x1002c39bc 0x100165184 0x10016471c 0x100329508 0x1004d5d24 0x1004f82e8 0x1816e94bc 0x1816e947c 0x1816eeb84 0x181c54d50 0x181c52bb8 0x181b7cc50 0x183464088 0x186e66088 0x1002d887c 0x18171a8b8)

Global Trace Buffer (reverse chronological seconds):
424518     libsystem_trace.dylib     0x000000018191c9cc dyld_image_header_containing_address(0x144f8e891) failed
908955     CFNetwork                 0x0000000182281a80 TCP Conn 0x144d93d80 SSL Handshake DONE
941332     CFNetwork                 0x0000000182281990 TCP Conn 0x144d93d80 starting SSL negotiation
941448     CFNetwork                 0x0000000182323f18 TCP Conn 0x144d77ac0 complete. fd: 14, err: 0
941738     CFNetwork                 0x0000000182323f18 TCP Conn 0x144d93d80 complete. fd: 10, err: 0
941917     CFNetwork                 0x0000000182325444 TCP Conn 0x144d77ac0 event 1. err: 0
942189     CFNetwork                 0x0000000182325444 TCP Conn 0x144d93d80 event 1. err: 0
953446     CFNetwork                 0x0000000182323f18 TCP Conn 0x144d55bb0 complete. fd: 5, err: 0
953773     CFNetwork                 0x0000000182325444 TCP Conn 0x144d55bb0 event 1. err: 0
957419     CFNetwork                 0x000000018232551c TCP Conn 0x144d93d80 started
961604     CFNetwork                 0x0000000182281a80 TCP Conn 0x144f07400 SSL Handshake DONE
029544     CFNetwork                 0x0000000182281990 TCP Conn 0x144f07400 starting SSL negotiation
037481     CFNetwork                 0x0000000182323f18 TCP Conn 0x144f07400 complete. fd: 7, err: 0
037481     CFNetwork                 0x000000018232551c TCP Conn 0x144d77ac0 started
043621     CFNetwork                 0x000000018232551c TCP Conn 0x144d55bb0 started
050324     CFNetwork                 0x0000000182325444 TCP Conn 0x144f07400 event 1. err: 0
076020     libsystem_trace.dylib     0x000000018191c9cc dyld_image_header_containing_address(0x144d7b291) failed
089100     libsystem_trace.dylib     0x000000018191c9cc dyld_image_header_containing_address(0x144d8e351) failed
132366     CFNetwork                 0x000000018232551c TCP Conn 0x144f07400 started
145189     CFNetwork                 0x000000018237fe5c Creating default cookie storage with default identifier
145189     CFNetwork                 0x000000018237fe28 Faulting in CFHTTPCookieStorage singleton
145189     CFNetwork                 0x00000001823d1e50 Faulting in NSHTTPCookieStorage singleton



Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
 libsystem_kernel.dylib        0x000000018183811c 0x18181c000 + 114972
 libsystem_pthread.dylib       0x0000000181904ef8 0x181900000 + 20216
 libsystem_c.dylib             0x00000001817a9dac 0x181748000 + 400812
 libc++abi.dylib               0x00000001812dd3f4 0x1812dc000 + 5108
 libc++abi.dylib               0x00000001812f9e98 0x1812dc000 + 122520
 libobjc.A.dylib               0x0000000181304248 0x1812fc000 + 33352
 libc++abi.dylib               0x00000001812f6f44 0x1812dc000 + 110404
 libc++abi.dylib               0x00000001812f685c 0x1812dc000 + 108636
 libobjc.A.dylib               0x0000000181304094 0x1812fc000 + 32916
 CoreFoundation                0x0000000181c9ec80 0x181b74000 + 1223808
Foundation                    0x0000000182624154 0x182580000 + 672084
MyAppName                     0x00000001005461f8 0x100024000 + 5382648
CoreFoundation                0x0000000181ca4a60 0x181b74000 + 1247840
CoreFoundation                0x0000000181b9c488 0x181b74000 + 165000
JavaScriptCore                0x00000001857ba294 0x18531c000 + 4842132
JavaScriptCore                0x00000001857b9e38 0x18531c000 + 4841016
JavaScriptCore                0x00000001857bb030 0x18531c000 + 4845616
JavaScriptCore                0x0000000185362ae8 0x18531c000 + 289512
JavaScriptCore                0x00000001857a1d38 0x18531c000 + 4742456
JavaScriptCore                0x000000018579bed8 0x18531c000 + 4718296
JavaScriptCore                0x00000001856c79fc 0x18531c000 + 3848700
JavaScriptCore                0x000000018536683c 0x18531c000 + 305212
JavaScriptCore                0x00000001853665ec 0x18531c000 + 304620
JavaScriptCore                0x0000000185756658 0x18531c000 + 4433496
MyAppName                     0x0000000100556ff0 0x100024000 + 5451760
CoreFoundation                0x0000000181ca2a1c 0x181b74000 + 1239580
CoreFoundation                0x0000000181ba0c5c 0x181b74000 + 183388
UIKit                         0x0000000186dfcc40 0x186de8000 + 85056
UIKit                         0x0000000186dfc844 0x186de8000 + 84036
MyAppName                     0x0000000100107044 0x100024000 + 929860
UIKit                         0x00000001872d3a50 0x186de8000 + 5159504
UIKit                         0x0000000187540b84 0x186de8000 + 7703428
MyAppName                     0x00000001002c50e4 0x100024000 + 2756836
MyAppName                     0x00000001002c39bc 0x100024000 + 2750908
MyAppName                     0x0000000100165184 0x100024000 + 1315204
MyAppName                     0x000000010016471c 0x100024000 + 1312540
MyAppName                     0x0000000100329508 0x100024000 + 3167496
MyAppName                     0x00000001004d5d24 0x100024000 + 4922660
MyAppName                     0x00000001004f82e8 0x100024000 + 5063400
libdispatch.dylib             0x00000001816e94bc 0x1816e8000 + 5308
libdispatch.dylib             0x00000001816e947c 0x1816e8000 + 5244
libdispatch.dylib             0x00000001816eeb84 0x1816e8000 + 27524
CoreFoundation                0x0000000181c54d50 0x181b74000 + 920912
CoreFoundation                0x0000000181c52bb8 0x181b74000 + 912312
CoreFoundation                0x0000000181b7cc50 0x181b74000 + 35920
GraphicsServices              0x0000000183464088 0x183458000 + 49288
UIKit                         0x0000000186e66088 0x186de8000 + 516232
libdyld.dylib                 0x000000018171a8b8 0x181718000 + 10424
MyAppName                     0x00000001002d887c 0x100024000 + 2836604
...

This is really a mess right? But don’t worry, we will figure it out step by step.

There are several values at the beginning, they are:

Process Information

Incident Identifier26D8A0C5-8B41-44F4-85A4-EAA1768C56A8

the crash id
CrashReporter Key383daadfbd4ac97cf2e53531fc349858fb09ae3c

the unique key that is related to crash and device information
Hardware ModeliPhone6,2

the device model, in this report it’s iPhone6
ProcessMyAppName [6390]

application name with crash process id
Path/private/var/containers/Bundle/Application/D093DF33-5DC7-4EE8-9F4B-0CDD98A41EE9/xxx.app/xxx

the application binary file location on device
Identifiercom.myCompany.MyAppName

application bundle id
Version1 (5.2.12)

application version
Code TypeARM-64 (Native)

application CPU architecture, this will be used in the following passage
Parent Processlaunchd [1]

parent process, normally it’s launchd cause iOS application is basically single-processed

Basic Information

Date/Time2016-06-14 09:43:43.43 +0800

time that application crashed
Launch Time2016-06-14 09:42:50.50 +0800

time that application opened
OS VersioniOS 9.3.2 (13F69)

device iOS version
Report Version105

crash log version

Exception

Exception TypeEXC_CRASH (SIGABRT)

the type of exception, there are several types and will be introduced later on
Exception Codes0x0000000000000000, 0x0000000000000000

the exception codes, which indicate the detailed information about exception
Exception NoteEXC_CORPSE_NOTIFY

detailed message about the exception
Triggered by Thread0

show thread that this crash occurred on

Thread Backtrace

All threads’ status and the backtrace at the crash moment are shown below. The thread which crash occurred on is marked with text Thread i Crashed, following by a list contains stack backtrace.

Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib        0x000000018183811c 0x18181c000 + 114972
1   libsystem_pthread.dylib       0x0000000181904ef8 0x181900000 + 20216
2   libsystem_c.dylib             0x00000001817a9dac
...

Common Crashes

Watchdog timeout
There is a description usually after exception information
```
 Application Specific Information：
 com.xxx.yyy　　 failed to resume in time
```
This means your application spent too much time on launching, perhaps there’re some network requests on main thread. This only happens in production environment, it’s ignored in development environment.
User force quit
Force quit does not mean kill background application, it’s when main thread is blocked and application has no response, user long-press power button and press home button to kill current application.
Notice that, device won’t generate crash log if application finished by user when it’s in background.
Low memory termination
When device does not have enough memory, and application memory usage does not decrease, this crash would occur.
Crash due to bugs
Crash created by program bug.

Common Exception Types

EXC_BAD_ACCESS - application read the forbidden memory
SIGSEGV : visit invalid address due to instance overrelease
SIGABRT : quit application because of an abort signal, normally it’s a Foundation lib protection mechanism
SEGV : invalid address, like empty pointer, uninitialized pointer, stack overflow
SIGBUS : read valid address but bus error
SIGILL : execute forbidden commands or have no privilege to execute
EXC_BAD_INSTRUCTION - thread execute forbidden commands
EXC_ARITHMETIC - divided by zero

Common Exception Codes

0xbaaaaaad - generally, this is not a crash, it’s just a system runtime status triggered by user’s unexpected operations.
0xbad22222 - activated from background too frequently
0x8badf00d - launching or resuming duration lasts too long
0xc00010ff - a large number of calculation for CPU and GPU, application termination due to device overheating
0xdead10cc - application still consume resources while it’s in background
0xdeadfa11 - force quit by user

Above are almost all elements in this file, however it’s still difficult for us to find out which line of code directly makes our application crashed. The most useful information I can get right now is the exception information, because I can see what kind of exception is thrown by our application.

Well, but I don’t know what code triggered this exception, so there must be a way to translate these backtrace, which is called Symbolization. but if you want, there’s still some tools can help you, like Xcode, Crashlytics, Flurry and so on.

In the rest of this passage, I’ll introduce how to use Xcode to symbolize the backtrace.

Xcode - Devices

Xcode offers an approach to fetch crash log from a iOS device and symbolize the crash information. With a translated crash report, it would be pretty easy to find out the error code.

Let’s start it.

To get the crash log on a specified device, you still need to connect your device to Mac, sync with iTunes. Then start Xcode if you finish synchronization, open Window -> Devices (or press ⇧ + ⌘ + 2) you’ll see the name of device that you just connected to your Mac is shown on left side of the window (if you cannot see it, just wait util sync finished), then click it.

You’ll get a view like this.

Press View Device Logs, and select All Logs. All recent crash logs of each application are shown below, choose one that you want to see, and the translated crash log displays on the right.

Sometimes you notice that there’s no difference between the crash log from Xcode and the crash log from user directory, that means Xcode cannot symbolicate this crash log backtrace. So How could this happen?

Symbolization

Symbolization is the procedure that translating stack backtrace which is shown as address format to line number of code, making the crash log more readable.

Symbolicating in LLDB

In the development environment, we can debug our program with breakpoint, see backtrace, see value of variable, execute script, etc. Sometimes we meet a crash (provided that we’ve set Global Exception Breakpoint), but Xcode locates in main.h file, thus we cannot know which line of code cause this exception on earth. There’re two ways to figure it out, one is opening debug navigator and select View Process by Thread, then you’ll see several threads below, usually our breakpoint stops on thread 1, spread thread 1 you will get a method call stack, select last one to see which method is invoked and trigger the crash. Another way is input bt command in console, it’s the same usage as former one.

A method call stack (backtrace) usually looks like this:

* thread #1: tid = 0x2466c, 0x000000010179b495 MyAppName`-[MyViewController viewDidLoad](self=0x00007fab44324e80, _cmd="viewDidLoad") + 85 at MyViewController.m:43, queue = 'com.apple.main-thread', stop reason = breakpoint 4.1
  * frame #0: 0x000000010179b495 MyAppName`-[MyViewController viewDidLoad](self=0x00007fab44324e80, _cmd="viewDidLoad") + 85 at MyViewController.m:43
    frame #1: 0x000000010596f984 UIKit`-[UIViewController loadViewIfRequired] + 1198
    frame #2: 0x000000010597593b UIKit`-[UIViewController __viewWillAppear:] + 120
    frame #3: 0x00000001059a5750 UIKit`-[UINavigationController _startCustomTransition:] + 1203
    frame #4: 0x00000001059b5b9b UIKit`-[UINavigationController _startDeferredTransitionIfNeeded:] + 712
    frame #5: 0x00000001059b6d0b UIKit`-[UINavigationController __viewWillLayoutSubviews] + 57
    frame #6: 0x0000000105b65503 UIKit`-[UILayoutContainerView layoutSubviews] + 248
    frame #7: 0x000000010588f980 UIKit`-[UIView(CALayerDelegate) layoutSublayersOfLayer:] + 703
    frame #8: 0x0000000105546c00 QuartzCore`-[CALayer layoutSublayers] + 146
    frame #9: 0x000000010553b08e QuartzCore`CA::Layer::layout_if_needed(CA::Transaction*) + 366
    frame #10: 0x000000010553af0c QuartzCore`CA::Layer::layout_and_display_if_needed(CA::Transaction*) + 24
    frame #11: 0x000000010552f3c9 QuartzCore`CA::Context::commit_transaction(CA::Transaction*) + 277
    frame #12: 0x000000010555d086 QuartzCore`CA::Transaction::commit() + 486
    frame #13: 0x000000010555d7f8 QuartzCore`CA::Transaction::observer_callback(__CFRunLoopObserver*, unsigned long, void*) + 92
    frame #14: 0x000000010392ac37 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #15: 0x000000010392aba7 CoreFoundation`__CFRunLoopDoObservers + 391
    frame #16: 0x00000001039207fb CoreFoundation`__CFRunLoopRun + 1147
    frame #17: 0x00000001039200f8 CoreFoundation`CFRunLoopRunSpecific + 488
    frame #18: 0x000000010a059ad2 GraphicsServices`GSEventRunModal + 161
    frame #19: 0x00000001057d4f09 UIKit`UIApplicationMain + 171
    frame #20: 0x0000000101a99fcf MyAppName`main(argc=1, argv=0x00007fff5e55e660) + 111 at main.m:14
    frame #21: 0x000000010939c92d libdyld.dylib`start + 1

If we only have the address of method, how do we know which line then? — The image command.

LLDB offers a command image to translate address to line number of code, here’s the usage:

image lookup --address 0x000000010179b495

and here’s what you got:

Address: MyAppName[0x00000001000fb495] (MyAppName.__TEXT.__text + 1020165)
Summary: MyAppName`-[MyViewController viewDidLoad] + 85 at MyViewController.m:43

Symbolicating with .dSYM file

Xcode
Before you gonna symbolicate crash log, there’re some files required.

.dSYM(Symbol) file
Every time build or archive application, Xcode would generate a version-related file to store symbol information, which is useful for debugging and symbolicating crash report.

The .dSYM file path: Find archive file : ~/Library/Developer/Xcode/Archives/(archive date)/xxx.xcarchive, then right click to show package content. the .dSYM file is located at dSYMs/MyAppName.app.dSYM.

binary file
the application binary file.

with these two files, Xcode will help you symbolicate the crash log.
Command
Usage: atos [-p pid] [-o executable] [-f file] [-s slide | -l loadAddress] [-arch architecture] [-printHeader] [address …]

a line of crash report looks like this:
```
  34  MyAppName                     0x0000000100165184 0x100024000 + 1315204
```
the architecture in crash log:
```
  Code Type:           ARM-64 (Native)
```
using following command to get the method name and line number
```
  atos -o MyAppName.app.dSYM/Contents/DWARF/MyAppName -arch arm64 -l 0x100024000 0x0000000100165184
```

references

iOS应用崩溃日志揭秘
 Symbolicating Crash Logs
(Stackoverflow) iphone: Where the .dSYM file is located in crash report
iOS Crash文件的解析（一）