Prototype for an RzIL based analysis chain

[Rot127]

Performance tests

• General runtime comparisons to old analysis. Slower is fine if the yield is bigger, but not slow for easily avoidable reasons.
• Analysing huge (3GB+) core dumps should never cancel due to out of memory or similar.

TODOs

• The statically known xrefs should not be an xref, but a successor address. One which shows under which condition the path was taken
• Rename the BB_CFG to block CFG to make the difference between basic and non basic block clear.
• Update docs (and diagrams).
• Extend xrefs to contain the access size in bits.
• Currently it only scans sections, not segments. But this fails for firmware binaries which might only have segments.
• Fix tests
• Basic blocks still nedd the instruction offsets added.

To merge

• Graph - performance #6382
• Add all call and other xrefs to the abl output #6269

Performance TODOs

...

Design changes

• On interpreter should be able to produce more than one abstract state.
  E.g. if the condition of a direct branch evaluates to a top value, it should produce a abstract state for each branch possibility.
  • This requires a central point which collects the state hashes. So we know when to terminate.
  • It also requires the interpreter to get a register/memory state to start out with.
  • Joining states is also not yet implemented (states are just hashed to check when an interpreter has reached a known state).
• Init all values of the VM with values according to the archs ABI. Should reduce number of false positive constants.
  • Can be implemented later with the memory model supporting stack, heap vars.
• Hashing the state is tricky. Only hashing the globals (registers) is not enough. If a program stores stuff on the stack. So if registers stay untouched the algorithm terminates too early.
  • There needs to be some configuration, what exactly belongs to the machine state and what not.
  • hash_state is super slow. It needs some faster way. Maybe hash the memory of the global hash table directly with sha (or other hw implemented hash functions?)
• We need an efficient way for the interpreter to store facts it observes during the execution of a basic block. E.g. if a certain register was read/written within a given scope.

For review start with

• rz_inquiry.h and rz_analysis.h for the API.
• Main interpretation loop is in rz_interpreter_run.
• interpreter_prototype.c is the plugin.
• rz_inquiry_interpreter gives an idea how the whole setup and API usage of an interpreter looks like.
• revng_function_detection.c to see the detection algo. But this one is only temporary.
• Run the analysis with aaaaIp <address> (non is current offset). The address is the first entry point executed by the interpreter.
  • It still prints a lot of not yet disabled warnings. So better run it with 2> /dev/null

tldr: interpreter

The interpreter simply collects all statically known branch targets.
Then starts interpreting at those addresses. It collects all yield the interpreter reports (one of them are xrefs).

After the interpreter finished (no new state reached) or failed. It then removes all addresses from the branch target vector which were covered by the interpreter (checks the xrefs it collected to do this).

Then picks the next branch target and starts interpreting from it again.

It continues until no branch targets are left.

tldr: function detection

The interpreter reports call candidates.
Those are instructions which might be calls (push address in the current basic block and branch).

The function detection iterates over all this call candidates and collects the called addresses.
These addresses are called "candidate function entry points (cfep)".

It iterates then over all cfep, adds a new function, and recurses into the basic block cfg, adding each basic block to the function.
Edges of call candidates are ignored.

Lastly, it converts the results to RzAnalysis objects.

Notes

Current design

This PR is only there to discuss design decisions for now.

The RzInquiry module combines for this prototype several roles. In the final version these roles are supposed to be in separated modules. So we can improve the performance independently.

Architecture implemented in this PR

  Modules                 Queues           Interpreter                       
 RzInquiry              Ring Buffers
┌───────────────────┐
│ ┌─────────────┐   │                      ┌─────────────────┐ 
│ │             │◄────────────────────────►│                 │
│ │             │   │       ┌─────────────►│ Interpreter 0   │
│ │ IO Handler  ├◄─────┐  ┌─│─────────────►│                 │
│ │             │   │  │  │ │              └─────────────────┘
│ │             ├◄─────┼──┼─┼─┐            
│ └─────────────┘   │  │  │ │ │                               
│                   │  │  │ │ │                               
│                   │  │  │ │ │                               
│ ┌─────────────┐   │  │  │ │ │                               
│ │             ├◄─────┼──┼─┘ │            ┌─────────────────┐
│ │             │   │  └──┼───┼───────────►│                 │
│ │ Yield       ├◄────────┼───┼───────────►│ Interpreter 1   │
│ │ Consumer    │   │  ┌──┼───┼───────────►│                 │
│ │             ├◄─────┼──┼─┐ │            └─────────────────┘
│ └─────────────┘   │  │  │ │ │                               
└───────────────────┘  │  │ │ │                               
                       │  │ │ │                   ...         
  ┌─────────────┐      │  │ │ │                               
  │             ├◄─────┼──┘ │ │
  │ RzIL Cache  │      │    │ │            ┌─────────────────┐
  │             ├◄─────┘    │ └───────────►│                 │
  │             │           └─────────────►│ Interpreter n   │
  │             │◄────────────────────────►┤                 │                                           
  └─────────────┘                          └─────────────────┘

Questions to discuss

• Added members RzILOpArgsVar->hash for performance.
• Use abbreviation intrpr or interp instead of interpreter to shorten name prefixes?
  • I went with interp.
• Where to implement the "get_il_bb"? In the Arch plugins? Or should it be a helper function in RzInquiry? Or the IL cache?
• How to design the filter of yield queues?
  • Pass function + const filter data to interpreters?
  • Filter on queue at receiving end? (Probably super bad for performance).
• Configuration of interpreter.
  • Via plugin specific config? Like the hexagon module.
  • Passed during initialization?
• Should all the interpreters be their own module? RzInquiry is then only the binding. But can they be a sub-module of RzInquiry?
  • Then what kind of plugins does RzInquiry expose?
  • This PR has the interpereters as sub-modules of RzInquiry.
• Should the interpreter implement RzILVM? I would argue no for some reasons:
  • The interpreter should simply implement RzAnalysisILVM. It is way simpler.
• The eval could only produce the delta between the old and new state.
  This would save a lot of unnecessary copies of registers which are never touched.
  This would mean we can never trow away any of the earlier states.

Performance

• The register file in an interpreter should have an access of O(1), cannot use allocations and doesn't run any hash functions. If we aim for performance our hash map implementation is too slow IMO, just by number of instructions executed (it is still worst case O(n) I think). Instead it would be nice to have all registers concatenated as one flat array. Indexed by a reg number.
• Add Queue implementation which works with shared memory instead of lists. This would save allocations as well.
• The basic block cfg should handle bb splitting and such on the go. Not in an O(|i2i| * |BBs|) function.

Function detection - Lessons learned

• Two CFGs can be generated. One with only statically know edges, one with edges discovered by the interpreters.
  • The interpreter CFG currently has BBs as nodes. The static CFG has instructions.
    Having BBs as nodes in the statuc graph is difficult to get. Because
    RzAnalysisOp.type are not as reliable as RzIL is. Maybe we can ignore this and
    just eat the inaccuracy?
  • This difference in node type means very expansive conversion.
    We really should get an r-tree implementation for that.
• CFG should be accessible via the API.
• CFG needs different kind of edges.
  • Normal branches.
  • Branch to unmapped address (invalid branches)
  • branch to "ignored" addresses (imported symbols and such).
  • Edges between calls an their return point (possibly not? Depends on our tail call detection).
• Different Basic Block node types
  • Normal basic block candidate
  • Unmapped basic block (imported, exported symbols and such).
• How to properly define the abstraction layers of basic blocks? Basic block (one entry, one exit), basic block candidate (one exit, possibly multiple entries, overlaps with other candidates), etc.
• Imported, exported, PLT stuff, ... symbols should be treated somehow special.
• Function Prelude's is very useful. We should have it available if possible.
• All discovered basic block candidates should be given to the interpreter.
• What ever the implementation does, it must be based on instruction packets for Sparc, Mips and Hexagon. Otherwise we never stop handling edge cases.
• The RzAnalysisOp.type is NOT a reliable source to distinguish jumps from calls or branches in general. op members should only be a source of static knowledge for the interpreter. The CFG detection should be based only on the interpretation results to generalize well.
• It's stupid to detect possible function entry points by assuming all outgoing edges from call candidates are going to an entry point. Its also not intuitive. The call candidate simply needs to store its target address it would call to.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.01%. Comparing base (c09c3b6) to head (750fd3b).

Files with missing lines	Patch %	Lines
librz/study/interpreter.c	0.00%	2 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
librz/study/interpreter.c	0.00% <0.00%> (ø)

... and 8 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c09c3b6...750fd3b. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.