1. Background

OpenPiton is an open-source research platform to evaluate novel microarchitectures — ranging from Network-on-Chips (NoCs) replacing traditional CPU buses to distributed L2 caches. Researchers are actively benchmarking common software and workloads on this platform to examine its performance and compatibility.

To simulate OpenPiton efficiently on workstations, researchers often rely on Verilator. Unlike traditional simulators that interpret Verilog line by line, Verilator compiles Verilog directly into highly optimized C++ classes, leveraging compiler-level optimizations to accelerate execution.

Yet, complex software like the Linux kernel can take weeks to boot, even in Verilator with the most aggressive optimization flags enabled.

2. Project Description

To address this bottleneck, my mentors, Dr. Guillem and Prof. Jonathan, proposed a novel approach to skip this boot latency: boot Linux to a ready state in QEMU, snapshot the memory state, and inject this checkpoint directly into the RTL simulation to resume execution.

architecture

Why QEMU?

We chose QEMU for the first phase because, even when cross-emulating via the TCG engine, it still outperforms Verilator by four to five orders of magnitude (roughly 10,000x to 100,000x).

The drastic speed distinction between QEMU and Verilator emerges from the fundamental difference in their abstraction levels:

  • QEMU (TCG) is an instruction emulator with a JIT engine which dynamically translates target assembly instructions to native host instructions, abstracting away hardware circuit details entirely.
  • Verilator is an RTL simulator, which must evaluate the exact logic state of every flip-flop, wire, and register on every single clock edge.

So offloading the billions of boot cycles to QEMU bypasses weeks of redundant RTL simulation in just a matter of seconds.

3. Technical Decisions

3.1 Hardware-Accurate Peripheral Emulation (openpiton-spike) vs. Runtime Address Relocation

Standard QEMU machine models (such as virt) use default peripheral addresses that are completely different with OpenPiton's physical memory map:

  • CLINT Base: 0xfff1020000
  • PLIC Base: 0xfff1100000
  • UART: 0xfff0c2c000

Because the Linux kernel and OpenSBI parse these base addresses from the Device Tree during early boot and cache them into internal driver pointers across physical memory, attempting to patch MMIO pointers post-capture is hard and fragile.

We built and maintained a dedicated openpiton-spike QEMU machine model whose MMIO layout, interrupt routing, and timer frequencies strictly mirror real OpenPiton RTL, ensuring 100% address fidelity between emulation and RTL.

3.2 Non-Intrusive M-Mode Restore Stub vs. Simulator Signal Injection

Instead of forcing internal register/wire signals through Verilator DPI handles, we place an M-mode assembly restore stub (restore_stub.S) at memory entry (0x80000000).

Forcing simulator internal signals ties the checkpoint format to specific Verilated netlist variable names, breaking compatibility whenever RTL is regenerated or refactored. The assembly restore stub executes natively on the simulated core, strictly obeying RISC-V architectural semantics (mret, PMP configuration) without simulator-specific dependencies.

Upon power-on reset, all harts jump to 0x80000000 to execute the restore stub in M-mode:

  1. Per-Hart Offset Resolution: Each hart indexes its own register blob via t6 = __reg_blob + (mhartid * BLOB_STRIDE).
  2. CLINT State Replay: Restores per-hart msip (IPIs) and mtimecmp, and hart 0 writes global mtime.
  3. PMP & CSR Restoration: Restores all 16 PMP regions and machine/supervisor CSRs (satp, stvec, sscratch, sepc, scause, stval, sie, mie, mcounteren, scounteren).
  4. FPU State Injection: Temporarily forces mstatus.FS = Dirty (3 << 13), restore f0-f31.
  5. Context Switch: Writes the authentic mstatus. Restores GPRs x1-x31, and executes mret directly into the checkpointed context.
stub-layout

3.3 Preserving Full OpenSBI Firmware for indispensable M-Mode Runtime

Rather than discarding OpenSBI or attempting to write a stripped-down bare-metal replacement, we explicitly preserve the entire initialized OpenSBI firmware resident at 0x80100000 in the memory image. This decision rests on three reasons:

  1. Strict RISC-V Privilege Separation: In RISC-V, Linux runs in S-Mode and is forbidden from directly accessing M-Mode CSRs. OpenSBI is the standard runtime provider that bridges OS requests to M-Mode operations.
  2. UART Console & Abstracted SBI Services: Preserving OpenSBI allows Linux to rely on SBI console ecalls (hvc0), timer management (sbi_set_timer), and inter-processor interrupts (sbi_send_ipi) through reliable firmware routines.
  3. Instruction Emulation & Illegal Instruction Handling: OpenSBI acts as a safety net by intercepting and emulating hardware-unimplemented instructions, preventing the kernel and userspace applications from crashing on unhandled illegal instruction traps.

Furthermore, by the time the kernel reaches our checkpoint trigger, OpenSBI has already finished its most expensive initialization phase. In the post-restore execution, OpenSBI functions purely as a lightweight, M-Mode service library. Writing a custom replacement would be immense effort with negligible performance benefit.

3.4 S-Mode Kernel Exception Breakpoint vs. U-Mode Userspace Breakpoint

Choosing where to halt execution during the fast QEMU boot phase is critical for checkpoint determinism.

In user space, programs reside in the lower half of the virtual address space (e.g., 0x10000). If a breakpoint is placed at a userspace address, earlier boot processes (especially init or early shell scripts) might accidentally hit that address before the intended workload is reached.

The Linux kernel resides in the upper half of the virtual address space (e.g., 0xffffffff80000000+), providing a globally unique mapping. By triggering the capture via a dedicated ebreak_trigger binary and configuring GDB to break on the kernel exception handler (do_trap_break), we ensure that:

  1. init and standard userland services will never ebreak and invoke do_trap_break during normal boot.
  2. The breakpoint fires only when our explicit trigger executes.
  3. The core halts with a clean kernel trap frame, allowing the kernel to naturally return to userspace.

Detailed Workflow

detailed-implementation

4. Milestones & Challenges

The project progressed through 3 phases, transitioning from toolchain familiarization to single-core implementation, and finally to SMP support and benchmark execution.

Phase 1: Toolchain Familiarization (Pre-GSoC Mar 2026 ~ Apr 2026)

Coming from a low-level software engineering background, I was initially not very familiar with hardware design workflows and the Verilog ecosystem. During which I identified and fixed two issues:

  1. GCC Toolchain: Building the bootrom failed under modern GCC. In newer RISC-V toolchain standards, zicsr and zifencei extensions were separated from the baseline rv64imac. Explicitly appending _zicsr_zifencei to -march resolved the build errors.

  2. PMP Entry Count: When building a combined memory image containing OpenSBI and arceOS (a lightweight Rust S-mode unikernel) and feeding it to Verilator, UART output remained completely blank.
    By cross-analyzing execution disassembly (dasm), OpenSBI sources, and OpenPiton RTL, I discovered that OpenSBI configured 15 Physical Memory Protection (PMP) entries for peripheral isolation, while CVA6 RTL defaulted NrPMPEntries to 4 (tile.v.pyv).
    Expanding hardware PMP entries from 4 to 16 in tile.v.pyv allowed OpenSBI to boot, and the arceOS banner appeared on the UART console shortly after.

Phase 2: Single-Core Implementation (Jun 2026 ~ Jul 2026)

With single-core bare-metal execution working on RTL, I began implementing the fast checkpoint-restore pipeline:

  1. Targeted QEMU Machine Model (openpiton_spike.c): I referenced QEMU's existing spike machine model while aligning peripheral addresses (CLINT, PLIC, UART) with OpenPiton's hardware map. QEMU successfully booted OpenSBI + arceOS.

  2. ISA Feature Sanitization: I attached GDB to QEMU, set a breakpoint at 0x80200000 (the entry of arceOS), extracted register/memory state, and embedded restore_stub.S at 0x80000000.
    However, running the image in Verilator caused another hang. Inspecting the disassembly revealed the core branched into _trap_handler_hyp — an unhandled trap for the RISC-V Hypervisor extension (H), which OpenPiton does not implement.
    QEMU's default -cpu rv64 enabled several extensions absent in CVA6 (including H, Sstc, Zicntr, Zihpm). Explicitly disabling them fixed the issue, and arceOS booted from the checkpoint on Verilator.

  3. Linux Bringup & Pipeline Automation: I then configured upstream generic Linux for OpenPiton with a minimal configuration:

    • Disabled unused PCIe, networking, block drivers, kernel modules, SMP, and KASLR to ensure memory location determinism and reduce simulation footprint.
    • Automated this capture-and-restore flow into scripts/extract.py and implemented convert.c in C to accelerate, establishing the core MinimumLinuxBoot repository.

Phase 3: SMP implementation & Benchmark Validation (Jul 2026 ~ Aug 2026)

Scaling from single-core to SMP introduced new challenges:

  1. Per-Hart Offset Calculation: In single-core mode, restore_stub.S restored register state from fixed memory offsets. On multi-core platform, all cores begin executing code at 0x80000000 simultaneously upon reset.
    To ensure each core accesses its own independent register state, I updated the stub to dynamically calculate its blob base address:
    base = __reg_blob + (mhartid * BLOB_STRIDE)
    where BLOB_STRIDE was expanded to 0x400 to accommodate the 792-byte register block.
mem-layout

  1. CLINT & CSR Wake-up Lockup: Initial multi-core boot attempts hung immediately after resume: one core was active while all others spun in loops without responding to interrupts after mret. Diagnosing the state revealed two missing links:

    • CLINT mtime/mtimecmp: The RTL CLINT resets mtime to 0, causing Linux's timekeeper to stall in timekeeping_advance(). Restoring mtimecmp[hartid] per core and global mtime from hart 0 resolved the timer deadlock.
    • mie Interrupt Enable: The core clears mie_q on reset, and its hardware WFI wake-up condition requires |(mip_q & mie_q). Without restoring mie, secondary cores parked in wfi could never be woken by IPIs. Restoring mie enabled normal SMP scheduling.

  2. Benchmark Validation & Simulation Concurrency Insights: I cross-compiled the NAS Parallel Benchmarks into the initramfs and validated them on smp=2, smp=4 passing all benchmarks (IS, MG, CG, EP) successfully.
    During multi-core benchmarking, my mentor Dr. Guillem pointed out that: Verilator simulation throughput does not simply increase with thread count like make -j.
    For instance, when simulating smp=2, setting Verilator --threads 2 is optimal. Too many threads will result in too many partitions of the simulation graph, wasting much time on inter-threads communication and synchronization.

5. Current Status

The core pipeline has been fully implemented and verified. Generating a bootable memory image requires only satisfying the dependencies described in the project readme and executing:

make capture -j$(nproc)

The pipeline automatically compiles the custom QEMU machine model, packages the bsc-linux, executes state extraction at the checkpoint trigger, and generates a mem.image ready for Verilator RTL simulation.

Below is the execution log of the NAS Parallel Benchmarks running on a 4-core OpenPiton setup resumed from a checkpoint:

[checkpoint] resumed
---------- is starting ----------


 NAS Parallel Benchmarks (NPB3.4-OMP) - IS Benchmark

 Size:  65536  (class S)
 Iterations:  10
 Number of available threads:  4



 IS Benchmark Completed
 Class           =                        S
 Size            =                    65536
 Iterations      =                       10
 Time in seconds =                     0.04
 Total threads   =                        4
 Avail threads   =                        4
 Mop/s total     =                    16.28
 Mop/s/thread    =                     4.07
 Operation type  =              keys ranked
 Verification    =               SUCCESSFUL
 Version         =                    3.4.4
 Compile date    =              09 Aug 2026

 Compile options:
    CC           = $(CROSS)gcc
    CLINK        = $(CC)
    C_LIB        = -lm
    C_INC        = (none)
    CFLAGS       = -O2 -fopenmp -march=rv64imafdc -mabi=lp64d
    CLINKFLAGS   = $(CFLAGS) -static


 Please send all errors/feedbacks to:

 NPB Development Team
 [email protected]


---------- is done ----------
---------- mg starting ----------


 NAS Parallel Benchmarks (NPB3.4-OMP) - MG Benchmark

 No input file. Using compiled defaults
 Size:   32x  32x  32  (class S)
 Iterations:                      4
 Number of available threads:     4

 Initialization time:           0.314 seconds

  iter   1
  iter   4

 Benchmark completed
 VERIFICATION SUCCESSFUL
 L2 Norm is  0.5307707005735E-04
 Error is    0.1681392571713E-12


 MG Benchmark Completed.
 Class           =                        S
 Size            =             32x  32x  32
 Iterations      =                        4
 Time in seconds =                     0.41
 Total threads   =                        4
 Avail threads   =                        4
 Mop/s total     =                    18.34
 Mop/s/thread    =                     4.59
 Operation type  =           floating point
 Verification    =               SUCCESSFUL
 Version         =                    3.4.4
 Compile date    =              09 Aug 2026

 Compile options:
    FC           = $(CROSS)gfortran
    FLINK        = $(FC)
    F_LIB        = (none)
    F_INC        = (none)
    FFLAGS       = -O2 -fopenmp -march=rv64imafdc -mabi=lp64d
    FLINKFLAGS   = $(FFLAGS) -static
    RAND         = randi8


 Please send all errors/feedbacks to:

 NPB Development Team
 [email protected]


---------- mg done ----------
---------- cg starting ----------


 NAS Parallel Benchmarks (NPB3.4-OMP) - CG Benchmark

 Size:        1400
 Iterations:                     15
 Number of available threads:     4

 Initialization time =           0.261 seconds

   iteration           ||r||                 zeta
        1       0.13844004790262E-12     9.9986441579140
        5       0.18512445023729E-14     8.5971549151767
       10       0.16933396286850E-14     8.5971775064409
       15       0.17533251786927E-14     8.5971775078648
 Benchmark completed
 VERIFICATION SUCCESSFUL
 Zeta is     0.8597177507865E+01
 Error is    0.1239725599088E-14


 CG Benchmark Completed.
 Class           =                        S
 Size            =                     1400
 Iterations      =                       15
 Time in seconds =                     3.62
 Total threads   =                        4
 Avail threads   =                        4
 Mop/s total     =                    18.39
 Mop/s/thread    =                     4.60
 Operation type  =           floating point
 Verification    =               SUCCESSFUL
 Version         =                    3.4.4
 Compile date    =              09 Aug 2026

 Compile options:
    FC           = $(CROSS)gfortran
    FLINK        = $(FC)
    F_LIB        = (none)
    F_INC        = (none)
    FFLAGS       = -O2 -fopenmp -march=rv64imafdc -mabi=lp64d
    FLINKFLAGS   = $(FFLAGS) -static
    RAND         = randi8


 Please send all errors/feedbacks to:

 NPB Development Team
 [email protected]


---------- cg done ----------
---------- ep starting ----------


 NAS Parallel Benchmarks (NPB3.4-OMP) - EP Benchmark

 Number of random numbers generated:        33554432
 Number of available threads:                      4

EP Benchmark Results:

CPU Time =     2.159
N = 2^   24
No. Gaussian Pairs =      13176389.
Sums =     1.051299420395299D+07    1.051517131857480D+07
Counts:
  0       6140517.
  1       5865300.
  2       1100361.
  3         68546.
  4          1648.
  5            17.
  6             0.
  7             0.
  8             0.
  9             0.


 EP Benchmark Completed.
 Class           =                        S
 Size            =                 33554432
 Iterations      =                        0
 Time in seconds =                     2.16
 Total threads   =                        4
 Avail threads   =                        4
 Mop/s total     =                    15.54
 Mop/s/thread    =                     3.89
 Operation type  = Random numbers generated
 Verification    =               SUCCESSFUL
 Version         =                    3.4.4
 Compile date    =              09 Aug 2026

 Compile options:
    FC           = $(CROSS)gfortran
    FLINK        = $(FC)
    F_LIB        = (none)
    F_INC        = (none)
    FFLAGS       = -O2 -fopenmp -march=rv64imafdc -mabi=lp64d
    FLINKFLAGS   = $(FFLAGS) -static
    RAND         = randi8


 Please send all errors/feedbacks to:

 NPB Development Team
 [email protected]


---------- ep done ----------
==============================================
 ALL BENCHMARKS DONE
==============================================

6. Proposals for Future Work

To further generalize and automate the MinimumLinuxBoot pipeline, several extensions can be explored:

  1. Arbitrary Execution State & Workload Snapshotting: Currently, checkpoints are triggered only at boot boundaries (such as do_trap_break right before workload execution).
    Expanding this framework to support on-demand snapshotting of arbitrary execution points.

  2. Matrix CI for Multi-Configuration Artifacts: Implementing a GitHub Actions build matrix will enable concurrent builds across diverse kernel configurations (e.g. varying SMP core counts, full debugging anchors) and separated userspace workloads (e.g. microbenchmarks, SPEC, NAS).

7. Deliverable

  • MinimumLinuxBoot Repository: Full booted bsc-linux memory image generation pipeline and automated GitHub Actions CI.
  • QEMU: Forked QEMU with custom openpiton machine type.
  • bsc-linux: BSC buildroot Linux, appended with ebreak_trigger utility.

8. Acknowledgements

This project greatly deepened my understanding of computer architecture and RISC-V systems, moving my perspective beyond the "Turing machine black box" familiar from my ICPC days into the concrete realities below the ISA.

I would like to express my deep gratitude to my GSoC mentors, Dr. Guillem Lopez-Paradís and Prof. Jonathan Balkind, for their invaluable guidance, technical discussions, and insightful advice on computer architecture and simulation methodologies.

I also thank the FOSSi Foundation and the Google Summer of Code program for providing this wonderful opportunity to contribute to open-source hardware research.