Approaches

Double Computation Approach

• talking about workloads

  • key insight: “piles”, reliance on the same cachelines is an issue
  • black box/white box -> reliability frontier
  • duplicate not protected compute, minimize costly I/O
  • cross/check because no executor uses overlapping resources etc.
  • workflow: scheduling (developer specifies dependencies)
    • for these workloads, blocked out

• open problem: hard-to-adapt interface requiring developer time limits adoption

• how do we make this more generally usable?

• Use case #3: Image processsing

  • Every block conflicts with N^2 other blocks

• Use case #2: Compression

  • each block needs the last 32KB of the preceding block

• Use case #1: Encryption

  • simplest: blocks are all independent of each other
  • radiation source side channel for encryption

• Key blockers

  • How do scheduling? Maybe don't use graph approach

• Metrics to examine

  • Fault tolerance (area x runtime)
  • Power consumption
  • Heat generation
  • Runtime overhead

• Use cases

  • Encryption - libtomcrypt on 1MB buffers
  • Packet processing - Pattern matching on packet buffers
  • Signals processing - FFTW (test both NEON/VFP)
  • Compression - 7zip extract
  • Machine Learning - OpenCV DNN?
  • Image processing - HCE program

• Graph coloring algorithms for scheduling

  • Most previous work use scheduling algorithms similar to bin packing
    • Undirected graphs aren’t really used to represent dependencies in MapReduce, etc.

• Directed graph of dependencies, with bidirectional arrows as conflicts?

  • Invert and create cliques vs k-coloring?
  • Not helpful, since we need to re-consolidate after each computation
    • Re-consolidation also helps us with checkpoint-and-restart

• Graph of conflicting items to account for continuous distributions

• n-coloring algorithm to distribute jobs

• 3 challenges at each stage

  • Job distribution: how to validate distribution algorithm? We might not even need to tho
  • Cache-aware TMR on compute
  • Checksum aggregation/minimizing runtime of vulnerable sections

• Apache Spark on Compute Devices

  • ~250MB overhead from JVM
  • ~500MB overhead from Apache Spark at idle
  • Might not be great for devices with only 2-4GB of RAM

• Examining system registers in QEMU

  • SP: immediately segfaults
  • LR: segfaults on return
  • PC: hangs or segfaults
  • CPSR: no effect (status register, maybe effects only show up on math computation)
  • FPSCR: no effect (FP status register, maybe effects only show up on math computation)
  • FPSID: no effect (config info register)
  • FPEXC: no effect (config info register)

• synchronizing compute across multiple cores

  • compute/store segments with comparison logic
  • Multithread with different cache access patterns works on Intel
    • 15% overhead in DMR vs single run
    • 20% overhead in TMR vs single run
    • 50% runtime improvement on multithread with cache clearing too?
  • But not on Snapdragon 801 CPU?
    • Same runtime between sequential and multithreaded

• Apache Spark

  • Reads/writes to/from memory in an immutable way, which might be aligned with what we want?

• New approach: duplicate compute and protect memory?

  • Compiler-Directed Differential Checksums - already done
  • Performance of this seems to be shit though, since you’ll need to update checksums at every step

• Register allocation types

  • OpenCV: Checked H,S,D,V vectors in OpenCV
    • Heavy-tailed distribution across all floating-point types
  • SPICE: Checked H,S,D vectors since compiled for ARM32
    • At most 8 used across all operations

• Testing HCE runtime with/without NEON

  • With NEON: 32 s
  • Without NEON: 174s
  • Vector processing may be required for some use cases

• Testing HCE with using 1/2 of all registers

  • No performance difference

Quantization Approach

• Real-world SPICE frame translation test

  • 24.27 seconds of runtime
  • Consistently ~700us difference between runtime of double/float
  • Flamegraph shows no difference between calculations in NEON/VFP

• see if we can hand-optimize matrix multiply and see perf?

  • Check if optimizer still writes to memory in for loop -> YES
  • Take a look at emitted LLVM bitcode and see if we can hand-rewrite it
  • LLVM passes

• How do we do the batch compute in a scheduled manner?

• Try matrix multiply with FP8 and see

  • Throughput does get better

• Make the matrix wider, do the multiplication twice

  • Fully optimized runs in about 8s, but the vectorization means it’s more vulnerable to radiation

• Test runtime with double with just compiler optimization

• Idea: duplicate instructions between different FPUs? SIMD/non-SIMD pipelines use different die areas so same performance and lesser risk

  • Timing results for VFP + NEON: 30s, vs 16s for VFP and 20s for NEON -> 6s savings over running it twice
  • Problematic for different architectures with different ISAs?

• WD: ECC on storage is done on each page, either in the disk controller or the driver

Symbolic Execution Approach

• test short-circuit vs non short-circuit runtimes
• replay-rerun of I/O intensive segments

Meeting Notes w/ Yaniv

• Estimated # of bit flips as a function of time
• Another idea: checksums on computation
  • Run angr on some function, recording inputs and outputs
    • Inputs can be any use that has not been def-ed in the segment
    • How to figure out what the output is?
    • Run symbolic execution and record the output state of code segment
      • Find some way to simplify the symbolic value of the output and use that as a "checksum"
    • Avoids the alias analysis issue entirely
  • If we have library calls, we can intercept them with a wrapper and record the inputs
  • Then when we have the outputs, validate them using some simplified function of the original input

Additional Thoughts

• Might want to explicitly target FSW offload/pure FSW use case
  • HotNets reviewers make a good point - HPC is not time-constrained
  • When we’re doing EDL or GNC time constraint could become a much bigger problem
• Next rad hardening meeting with NASA will talk about a DSL for flight software as well

Software ECC

• parity bits location will be important
• benchmark heat generation from memory ECC
• understand challenges more
  • when do read and write checksums?
  • how to efficiently implement erasure algorithm in software?
• how to motivate use case, since ECC memory is cheap already?
• is lower-precision good enough? if so why not
• how good is hardware support for ECC memory? i.e. how does ECC respond to radiation errors
• summarize reviews in a google sheet (group by concern)

Memory Scrubber

Lots of kernel hacking to get it working How do we send a page to memory to verify it? Does a memcpy for each page make sense? Getting a JPL intern to do the work? Need to wait on NSPIRES grant