On a Thursday morning our team shipped version 4.12 of a grocery delivery app to 20% of Android users and through phased release on iOS. By the afternoon, crash-free users on Android had dropped from 99.6% to 98.9%. The top new issue in our crash reporter looked like this:

Fatal Exception: java.lang.NullPointerException
Attempt to invoke virtual method 'java.lang.String a.b.c.d()' on a null object reference
    at a.b.f.e(SourceFile:3)
    at a.b.g.onBindViewHolder(SourceFile:41)

Nobody could tell what a.b.f.e was, and the iOS side had a similar problem: a cluster of crashes showing only hex addresses inside our binary because the dSYM for that build had never been uploaded. It took us four hours to get readable traces, and only then did the actual debugging start.

This guide is the workflow we built afterwards so that does not happen again. It covers six stages: collecting crashes, symbolication on iOS and Android, grouping, reproduction, fixing, and tracking the fix across releases. By the end you should be able to set up a pipeline where every crash arrives readable, grouped and tied to a release you can halt.

Stage 1: collect crashes from more than one source

Use a crash reporting SDK (Firebase Crashlytics, Sentry, Bugsnag or similar) as the primary source, because it adds breadcrumbs, custom keys and user counts. Also watch the platform consoles:

  • Google Play Console, Android vitals shows crashes and ANRs, including those that happen before your SDK initialises. Google publishes bad behaviour thresholds for user-perceived crash and ANR rates, and exceeding them can reduce your app's visibility on Play.
  • Xcode Organizer and App Store Connect collect crash logs from users who opted in to share analytics with developers, including some crashes in app extensions and early launch.

Agree on one headline metric. We use crash-free users per release version, because crash-free sessions can look healthy while a crash that hits a single critical screen locks out a specific group of people.

Stage 2a: symbolication on iOS

An iOS crash report contains memory addresses. To turn them into function names and line numbers you need the dSYM (debug symbols) produced by the exact build that crashed, matched by its UUID.

Make sure release builds produce dSYMs. In the target's build settings, Debug Information Format must be DWARF with dSYM File for the Release configuration. Since Xcode 14 deprecated bitcode, the App Store no longer recompiles your app, so the dSYMs from your own archive are the ones that matter; there is nothing to download back from App Store Connect for new builds.

Check which UUIDs a dSYM contains and compare them with the crash report's binary images section:

dwarfdump --uuid MyApp.app.dSYM

UUID: 6F1C2E0A-9B3D-3C71-A6E2-4D5B9A1F0C77 (arm64) MyApp.app.dSYM/Contents/Resources/DWARF/MyApp

To symbolicate a single address by hand, use atos with the load address of the binary from the crash report:

atos -arch arm64 \
  -o MyApp.app.dSYM/Contents/Resources/DWARF/MyApp \
  -l 0x104a2c000 0x104a3f1b4

-[CartViewController applyPromotion:] (in MyApp) (CartViewController.m:212)

In practice you upload dSYMs from CI so the crash reporter does this automatically. With Crashlytics that is the upload-symbols tool shipped with the SDK; with Sentry it is sentry-cli debug-files upload. Upload dSYMs for every framework you build yourself too, not only the main app binary.

Stage 2b: symbolication on Android

Release builds usually run R8, which shrinks and obfuscates code. The mapping from obfuscated names to the original ones is written to app/build/outputs/mapping/release/mapping.txt. Every build has its own mapping file, so you need the one from the exact build that crashed.

For one-off traces, the retrace tool from the Android SDK command-line tools does the conversion:

retrace app/build/outputs/mapping/release/mapping.txt crash.txt

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String com.example.cart.Promotion.getLabel()' on a null object reference
    at com.example.cart.CartSummaryFormatter.format(CartSummaryFormatter.kt:42)
    at com.example.cart.CartAdapter.onBindViewHolder(CartAdapter.kt:118)

For the crash reporter, the Crashlytics Gradle plugin uploads the mapping file during the release build by default when minification is enabled. For Play Console, upload the deobfuscation file for each release, or confirm in App bundle explorer that it was picked up with your bundle.

Native code (NDK, game engines, many media SDKs) needs separate symbols:

// app/build.gradle.kts
android {
    buildTypes {
        release {
            ndk { debugSymbolLevel = "FULL" } // packages native symbols for Play Console
        }
    }
}

Crashlytics also needs native symbols uploaded, controlled by nativeSymbolUploadEnabled in the firebaseCrashlytics block. For a raw tombstone, ndk-stack symbolicates locally:

ndk-stack -sym app/build/intermediates/merged_native_libs/release/out/lib/arm64-v8a -dump tombstone.txt

Cross-platform frameworks add their own layer. Flutter builds with --obfuscate --split-debug-info need the symbol files kept per build, and flutter symbolize reads them. React Native needs the JavaScript source maps for each release uploaded to the crash reporter.

Stage 2c: make symbol upload a release gate

All of the above fails silently if one step is skipped. We added a CI step that runs after the release build and fails the pipeline if symbols are missing:

# iOS: every UUID in the archive must be known to the crash reporter
for dsym in build/MyApp.xcarchive/dSYMs/*.dSYM; do
  dwarfdump --uuid "$dsym" | awk '{print $2}'
done > uuids.txt
./scripts/check-dsyms-uploaded.sh uuids.txt   # queries the crash reporter's API

# Android: mapping file must exist and be archived with the build
test -s app/build/outputs/mapping/release/mapping.txt
cp app/build/outputs/mapping/release/mapping.txt "artifacts/mapping-${VERSION_CODE}.txt"

We archive every mapping file and dSYM with the build artifacts for at least as long as the version is installed anywhere. A crash from a two-year-old version still needs its own symbols.

Stage 3: group and prioritise

Crash reporters group by stack trace, usually the top frames in your own code. Grouping goes wrong in two common ways:

  • Over-grouping: generic frames such as abort(), objc_exception_throw or a shared error helper sit at the top of many unrelated crashes. Configure the reporter to ignore those frames for grouping, or set a custom fingerprint in the SDK for errors raised from a common helper.
  • Under-grouping: the same bug appears as several issues because line numbers changed between releases or the crash happens on different threads. Merge them manually and add a note.

Prioritise with three questions. How many users does it affect in the current release? Is it new or rising since the last release? Does it hit a critical flow such as login, checkout or payment? A crash affecting 0.3% of users on the checkout screen outranks a 1% crash in a settings page few people open.

Stage 4: reproduce with context

A readable stack trace tells you where. Context tells you why. Before crashes happen, make sure the SDK records:

  • Breadcrumbs: screen views, key taps and network calls with status codes, not bodies.
  • Custom keys: feature flag variants, A/B test groups, account type, app state (foreground, background, restored after process death).
  • Device and OS distribution: the reporter shows it per issue. In our 4.12 crash, 100% of events had the promo_v2 flag enabled, which pointed us at a promotion without a label from the new backend.

Then reproduce deliberately: same flag, same OS version, same data. On Android, "Don't keep activities" in developer options and adb shell am kill <package> while backgrounded reproduce many state-restoration crashes. On iOS, the memory graph debugger and Address Sanitizer catch many EXC_BAD_ACCESS crashes before users do.

Stage 5: fix with a test

Write the failing test first, from the crash data. For our null label:

@Test
fun `formats cart when promotion has no label`() {
    val cart = cartWith(promotion = Promotion(id = "p1", label = null, discountMinor = 500))
    assertEquals("-5.00", CartSummaryFormatter().format(cart).discountText)
}

Fix the cause, not only the crash site. Here, the formatter treated the label as non-null even though the API contract allowed null; we fixed the Kotlin model to String? and handled the missing case, and the backend team added a default label.

Stage 6: track the fix per release

A fix is not done until the release that contains it is shown to be healthy:

  • Stage the rollout. Google Play staged rollouts let you release to a percentage of users and halt the rollout if crash rates jump. App Store phased release spreads an automatic update over seven days and can be paused.
  • Compare by version. Watch crash-free users for the new version against the previous one at the same rollout age, not against last month's average.
  • Close the loop. Mark the issue as resolved in the version that ships the fix. Good crash reporters reopen it as a regression if it appears again in a later version.

For 4.12 we halted the Android rollout at 20%, shipped 4.12.1 the next morning, and crash-free users for 4.12.1 reached 99.7% after 48 hours. The iOS phased release continued: once its dSYM was uploaded, the iOS cluster turned out to be an older, already-known issue rather than a 4.12 regression.

Common mistakes

Mistake Why it hurts Do this instead
Uploading symbols manually Someone forgets, and that release is unreadable forever Upload from CI and fail the build if missing
Keeping only the latest mapping file Older versions in the wild cannot be deobfuscated Archive mapping files and dSYMs per build
Tracking one crash-free number for all versions Hides regressions in new releases Compare per version at equal rollout age
Releasing to 100% at once No time to react to a new crash Use staged rollouts and phased releases
Logging request bodies as breadcrumbs Sends personal data to a third party Log routes, status codes and IDs only

Key takeaways

  • A crash report is only useful once it is symbolicated; treat symbol upload as part of the release.
  • dSYMs and R8 mapping files are tied to one exact build; archive them per version.
  • Tune grouping so one issue equals one bug, then prioritise by users, trend and flow.
  • Capture breadcrumbs and custom keys before you need them; they shorten reproduction the most.
  • Ship fixes with a test and confirm them per release with staged rollouts.

References