Benchmarking DeepDiff Against BinDiff on Stripped and Cross-Compiler Binaries

DeepDiff matches related functions across binaries. This matching layer can support patch verification, version analysis, firmware comparison, malware research, and other work that requires code alignment.

In our earlier DeepDiff post, we introduced the product through a firmware patch-verification example. This post focuses on a narrower question: how accurately can DeepDiff match functions when two binaries no longer look alike?

All of these tasks start with the same question:

Which function in one binary matches a function in the other binary?

This sounds simple when both binaries come from the same build system. In that case, addresses may move, but much of the code keeps the same shape.

The problem becomes much harder when the compiler changes, symbols are removed, libraries are linked into the program, or code moves from one function to another. The two functions may come from the same source and still look very different at the binary level.

To answer that question, we compared DeepDiff with Google BinDiff across 21 binary pairs. The tests cover three common problems in binary matching:

1. Different compilers can create very different control-flow graphs.
2. A diffing tool cannot match functions that its disassembler never found.
3. A version update can move code to a function with a different name.

This is a benchmark of DeepDiff's core matching layer. It is not a test of one specific workflow built on top of those matches.

What We Tested

We tested 21 binary pairs:

• 6 Lua pairs built with GCC or Clang at O2 or O3
• 15 minigzip pairs built with different zlib versions

Both tools received stripped binaries. To check their results, we used matching unstripped binaries with symbols.

We counted a match as correct when both addresses had the same symbol name. For example, if a tool produced this match:

0xf610 -> 0xee50

and both addresses had the symbol name luaD_call, we counted it as correct. Some tool output used rebased addresses, so we converted those addresses back to ELF virtual addresses before checking them.

We used two standard measures:

• Precision: Of all matches returned by the tool, how many were correct?
• Recall: Of all expected function matches, how many did the tool find?

Across all 21 pairs, the combined results were:

Tool	Precision	Recall	Correct matches	Wrong matches
DeepDiff	0.892	0.879	5,171	626
Google BinDiff	0.757	0.642	3,780	1,215

DeepDiff found more correct matches and made fewer wrong matches in these tests. The reason was not one single feature. Each test exposed a different part of the matching problem.

Finding 1: The Same Source Can Produce a Different Graph

BinDiff uses structural signals such as function graphs, basic-block graphs, call graphs, and edge patterns. These signals work well when the two binaries have similar structures.

That condition often holds when both binaries use the same compiler. In our Lua test, BinDiff reached about 0.95 precision and recall when both sides were built with Clang.

The result changed when one side used GCC and the other used Clang. The table below shows the average result in both directions for each pair:

Pair	BinDiff precision / recall	DeepDiff precision / recall
Clang O2 vs. Clang O3	0.948 / 0.941	0.964 / 0.951
GCC 11 O2 vs. GCC 13 O3	0.810 / 0.793	0.861 / 0.898
GCC O2 vs. Clang O2	0.589 / 0.579	0.844 / 0.838
GCC O2 vs. Clang O3	0.590 / 0.569	0.885 / 0.834
GCC O3 vs. Clang O2	0.598 / 0.597	0.842 / 0.850
GCC O3 vs. Clang O3	0.635 / 0.629	0.865 / 0.813

Across the GCC-to-Clang pairs, BinDiff fell to about 0.59. DeepDiff stayed between about 0.84 and 0.89.

One pair makes the difference clear. The lua-clang-O2 -> lua-O2 ground truth contained 659 functions:

Tool	Correct matches	Wrong matches	Precision	Recall
DeepDiff	552	102	0.844	0.838
BinDiff	380	265	0.589	0.577

For this pair, DeepDiff found 172 more correct matches and made 163 fewer wrong matches.

When Graph Size Changes

The Lua function luaH_getn shows how large the compiler difference can be:

Build of luaH_getn	Basic blocks	Edges	Instructions
Clang O2	86	137	298
GCC O2	46	54	137

Both versions came from the same source function. The Clang version had more than twice as many instructions as the GCC version.

BinDiff gave the correct luaH_getn -> luaH_getn match a similarity score of only 0.104. It matched Clang's luaH_getn to GCC's close_func instead.

DeepDiff found the correct match:

0x1d9e0 -> 0x1b280, similarity 0.826

When Graph Size Looks Similar

Large graph changes are not the only problem. The two versions of luaD_call were close in size:

• Clang: 8 basic blocks and 42 instructions
• GCC: 7 basic blocks and 40 instructions

BinDiff still matched luaD_call to dothecall. DeepDiff matched it correctly:

0xf610 -> 0xee50, similarity 0.937

The image below shows BinDiff's match. In the stripped binary, sub_F610 is luaD_call and sub_10D80 is dothecall.

This is an important failure case. Many small functions have similar graph shapes. When the compiler changes code layout, graph structure can point to the wrong nearby function even when the number of blocks and instructions looks close.

More Examples from Lua

On this one Lua pair, DeepDiff correctly matched 220 functions that BinDiff matched incorrectly or did not match.

Here are several examples:

Function	DeepDiff match	DeepDiff similarity	BinDiff result
luaD_call	0xf610 -> 0xee50	0.937	matched to dothecall
luaD_precall	0xf2d0 -> 0xeaf0	0.916	matched to luaT_getvarargs
luaD_pcall	0xfb60 -> 0xf2f0	0.934	matched to luaD_call
luaH_getn	0x1d9e0 -> 0x1b280	0.826	matched to close_func
luaH_next	0x1bda0 -> 0x19f50	0.959	matched to luaG_errormsg
luaH_size	0x1c580 -> 0x1a8f0	0.997	not matched
luaG_errormsg	0xd680 -> 0xd460	0.975	matched to luaG_addinfo
luaT_callTM	0x1e140 -> 0x1b710	0.965	matched to gmatch
luaS_resize	0x1af00 -> 0x192e0	0.903	matched to luaD_checkminstack
luaO_pushfstring	0x15990 -> 0x14290	0.995	matched to luaK_concat

Two more graph views show the same pattern.

BinDiff matched luaD_pcall to luaD_call. DeepDiff matched luaD_pcall correctly at 0xfb60 -> 0xf2f0.

BinDiff also matched luaG_errormsg to luaG_addinfo. DeepDiff found the correct luaG_errormsg match at 0xd680 -> 0xd460.

BinDiff labels stripped functions with names such as sub_XXXX. The function names above come from the symbol-based ground truth used only for checking the results.

Finding 2: You Cannot Match a Function You Never Found

The Lua test was mainly about matching quality. The minigzip test exposed a different problem: function coverage.

BinDiff works from functions identified by IDA. In a stripped, statically linked binary, IDA may not identify every library function. This is common for linked functions that the main program does not call.

For minigzip64-1.2.11:

• The original binary with symbols had 133 function symbols.
• Versions 1.2.11 and 1.2.12 had 130 functions in common.
• IDA identified 138 function nodes in the stripped binary.
• Only 76 of the 130 expected functions were present in IDA's function list.

The total count of IDA functions was not the important number. IDA also found functions outside our symbol-based ground truth. The key result was that 54 of the expected functions were missing from the input given to BinDiff.

For minigzip64-1.2.11 -> minigzip64-1.2.12, the results were:

Tool	Correct matches	Wrong matches	Precision	Recall
DeepDiff	121	9	0.931	0.931
BinDiff	76	0	1.000	0.585

BinDiff was correct for every function it matched. This is a strong result. But it found only 76 of the 130 expected matches because the other functions were not available to it.

DeepDiff found 47 of the 54 functions that BinDiff missed. Examples include:

Function	Side 1 address	DeepDiff matched address	DeepDiff similarity	BinDiff
gzopen	0x2200	0x3210	1.000	missed
gzbuffer	0x22c0	0x32d0	1.000	missed
gzseek64	0x23e0	0x33f0	1.000	missed
deflateBound	0x6da0	0x7d10	1.000	missed
deflateSetHeader	0x6ae0	0x7a20	1.000	missed
inflateCopy	0xc4a0	0xd460	1.000	missed
zlibCompileFlags	0x10100	0x11110	1.000	missed
zError	0x10110	0x11120	0.985	missed
adler32_combine	0x10830	0x11840	1.000	missed
main.cold	0x143a	0x243a	1.000	missed

Many of these functions were almost identical in the two zlib versions. The difficult part was not deciding whether the functions matched. The difficult part was finding the function boundaries in the first place.

gzopen is a simple example:

1. The symbol table placed gzopen at 0x2200.
2. IDA did not create a function at 0x2200.
3. DeepDiff still found its match at 0x3210 in the next version.

This coverage problem explains most of the recall difference in this minigzip pair. It is separate from the GCC-to-Clang result, where both tools had the functions but differed in how well they matched them.

Finding 3: The Name Can Stay While the Code Moves

Most minigzip version pairs were close. The largest change appeared in zlib 1.3.2.

In version 1.3.2, several public API functions became small tail-call wrappers. Their main code moved into another function, often one with a *64, _z, or _gen64 name.

The symbol sizes make the change easy to see:

Function	1.3.1 size	1.3.2 size	Change
gzseek	497	9	became a wrapper
gzseek64	497	520	now contains the main code
deflateBound	472	9	became a wrapper
deflateBound_z	-	647	now contains the main code
crc32_combine	247	9	became a wrapper
crc32_combine_gen64	167	192	contains shared code
gzgetc_	156	9	became a wrapper
gzgetc	156	188	now contains the main code

This creates a problem for any result checked only by symbol name. The old symbol can still exist, but it no longer contains the old code. A useful binary match should help the analyst follow the code to its new location, even when the name changes.

For older zlib versions compared with 1.3.2, the results were:

Tool	Correct matches	Wrong matches	Precision	Recall
DeepDiff	about 110-112	about 11-20	0.85-0.91	about 0.86
BinDiff	about 65-71	about 6-11	0.855-0.922	about 0.50

For 1.3.1 -> 1.3.2, the strict symbol-name check counted 11 wrong matches for each tool. The number was the same, but the errors were not the same kind:

Tool	Name-based wrong matches	Mean similarity	Matches with similarity >= 0.90
DeepDiff	11	0.978	11 / 11
BinDiff	11	0.537	1 / 11

Nine of DeepDiff's 11 name-based errors followed code to its new function:

• gzseek in 1.3.1 matched gzseek64 in 1.3.2.
• deflateBound in 1.3.1 matched deflateBound_z in 1.3.2.
• crc32_combine in 1.3.1 matched crc32_combine_gen64 in 1.3.2.

The strict check marked these as wrong because the names differed. For an analyst, these are useful matches: they show where the code moved.

The other two DeepDiff matches were clear errors:

• adler32_combine64 -> adler32_combine
• deflatePending -> deflateUsed

BinDiff's wrong matches were mostly low-similarity matches between unrelated functions:

Function in 1.3.1	BinDiff match in 1.3.2	Similarity
gzputc	gzgetc	0.739
inflateInit2_	gz_look	0.418
gz_look	gz_error	0.352
gzprintf	gzvprintf	0.289
gz_error	deflateReset	0.171
gzgetc_	gzprintf	0.067

The zlib 1.3.2 case shows why binary matching is more than checking whether two symbols have the same name. Sometimes the most useful result is the function with a different name but the same code.

What This Means for Binary Analysis

The benchmark shows where DeepDiff's matching layer is most useful.

When the compiler changes, DeepDiff can still match functions even when their control-flow graphs have very different sizes and shapes.

When a stripped static binary contains functions that IDA did not identify, DeepDiff can find many of those function boundaries and include them in the comparison.

When an update moves a function's main code behind a wrapper, DeepDiff can follow that code to a new symbol instead of stopping at the old name.

Release-to-release comparison is one use of this alignment, but it does not define the product. The same matching layer can support any analysis that needs to connect related code across two binaries. The analyst's question decides how those matches are used.

What This Test Does Not Show

There are a few limits worth stating clearly.

• This is not a test of every binary type. The test set contains Lua and minigzip binaries. Other programs, compilers, platforms, and forms of obfuscation may produce different results.
• The symbol check is strict. It treats different symbol names as a wrong match, even when code moved and the result is useful. We reviewed those cases separately instead of changing the scoring rule after the test.
• The minigzip gap includes function coverage. Much of DeepDiff's recall gain came from finding functions that were missing from IDA's function list. This is different from the cross-compiler Lua result, which tested matching quality.
• BinDiff is strong when its input is complete and the binaries are close. It reached 1.000 precision on the detected functions in unchanged zlib pairs. Our result is not that BinDiff performs poorly in every setting.

The narrower result is more useful:

In these tests, DeepDiff was more reliable when the binaries were built with different compilers, when stripped static code was missing from IDA's function list, and when an update moved code between functions.

These cases are common in real binary analysis. The results give measured evidence for DeepDiff's core purpose: finding related code across binaries even when compiler choices, missing function boundaries, or code movement make the match difficult.