Skip to content

Chiplet Disaggregation + UCIe (v5)

v5 splits the v4 monolithic system_top into three dies connected by modeled UCIe (Universal Chiplet Interconnect Express) links. The external system_top port list is unchanged — cocotb still drives the same chip boundary — but internally, what used to be same-module wiring is now flit traffic crossing die boundaries with credit-based flow control and fixed crossing latency.

This mirrors real chiplet packages (AMD CCD/IOD, Intel tile splits): compute dies stay "dumb" from an I/O standpoint, and a single base/I-O die aggregates shared memory and all external I/O.

Why split it, and why UCIe raw mode

Duck Server's inter-die traffic is a custom instruction/result stream, not memory-mapped PCIe TLPs or CXL.mem/cache semantics — UCIe's raw (streaming) mode is the right fit: the protocol layer is fully custom, carried over the standard UCIe die-to-die adapter + physical layer bring-up sequence. It's also the natural next step for a project already modeling a heterogeneous SoC — CPU and NPU compute stay on their own dies, the base die owns the network ingress, MOESI L2, and external DRAM/DMA, and interconnect behavior (credits, training, crossing latency) becomes an explicit, inspectable model instead of implicit same-cycle wiring.

Each ucie_chan instance carries exactly one flit type in one direction — five dedicated channels total instead of a generic N-VC crossbar — because Duck Server's actual traffic classes are few and well-known ahead of time (see the flit table below). This is the same idea real UCIe virtual channels solve (avoid head-of-line blocking between unrelated traffic), sized down to fit.

Package Diagram

                 ┌─────────────────────────────────────────────┐
                 │              base_die_top                    │
                 │                                               │
  net_valid/     │   ┌───────────┐        ┌──────────────────┐  │
  instruction ──►│   │ noc_router│        │ duck_coherency_ctrl│  │
  net_ready  ◄───│   └─────┬─────┘        │  (MOESI L2)        │  │
                 │         │              └─────────┬─────────┘  │
                 │   cpu_accept            cpu_read/write/addr   │
                 │         │              (via mem_req/mem_rsp    │
                 │         │               bridge FSM)            │
                 │         │                        │             │
                 │         │              dram_*    │  npu_dma_*  │
                 └─────────┼────────────────┬───────┼─────────────┘
                           │                ▼        (top-level ports)
              instr_load   │           External DRAM
              (base→cpu)   │  npu_disp (base→npu)
                    mem_req│  npu_result (npu→base)
                mem_rsp    │       │
                    │      │       │
         ┌──────────▼──────▼───┐   │      ┌──────────────────────┐
         │  ucie_link_ctrl       │   │      │  ucie_link_ctrl        │
         │  (CPU link training)  │   │      │  (NPU link training)   │
         └──────────┬────────────┘   │      └───────────┬────────────┘
                     │  cpu_link_up   │        npu_link_up│
         ┌───────────▼────────────────▼──────────────────▼───────────┐
         │   3× ucie_chan (CPU link)      2× ucie_chan (NPU link)     │
         │   instr_load / mem_req / mem_rsp   npu_disp / npu_result   │
         └──────────┬──────────────────────────────────┬─────────────┘
                     │                                  │
          ┌──────────▼─────────────┐          ┌─────────▼───────────────┐
          │   cpu_chiplet_top       │          │   npu_chiplet_top        │
          │  ┌───────────────────┐ │          │  ┌────────────────────┐ │
          │  │ imem (256 words)  │ │          │  │ 16-entry result FIFO│ │
          │  └─────────┬─────────┘ │          │  └──────────┬─────────┘ │
          │            ▼           │          │             ▼           │
          │      riscv_core         │          │           npu            │
          │  (unmodified)           │          │      (unmodified)        │
          │      halt ──────────────┼─► top-level halt (unchanged pin)   │
          └─────────────────────────┘          └──────────────────────────┘

noc_router.sv, duck_coherency_ctrl.sv, riscv_core.sv, and npu.sv are all unmodified — every *_chiplet_top.sv / base_die_top.sv file is a wrapper that translates that sub-module's native level-held handshakes into flit traffic on its side of a link.

The Three Dies

File Die Contains
base_die_top.sv Base / I-O die noc_router, duck_coherency_ctrl (MOESI L2), all external I/O (network, DRAM, NPU DMA, NPU result bus), plus the two link-facing bridge adapters
cpu_chiplet_top.sv CPU chiplet riscv_core, the 256-word imem front end, halt detection, the mem_req/mem_rsp bridge FSM
npu_chiplet_top.sv NPU chiplet npu (16×16 MAC array), a 16-entry result buffer + drain FSM

Why does the base die keep the L2/coherency controller instead of the CPU? Because the NPU DMA port also needs coherent access to the same memory — keeping MOESI on the base die means both compute dies reach it the same way (CPU over the mem_req/mem_rsp UCIe channels, NPU DMA over its own top-level port), rather than routing NPU DMA traffic through the CPU chiplet.

Two physical links, five channels total:

Link Channel Direction Depth Carries
CPU link instr_load base → cpu 256 (matches imem) Accepted network instruction word
CPU link mem_req cpu → base 2 (single outstanding) Load/store request (addr, data, byte_en, type)
CPU link mem_rsp base → cpu 2 Load/store completion (rdata)
NPU link npu_disp base → npu 1 (mirrors original npu_ready) Dispatch instruction (custom-1 word)
NPU link npu_result npu → base 4 (absorbs output bursts) One accumulator result (tag = row index)

Flit opcodes and channel depths live in rtl/duck_ucie_pkg.sv (FLIT_* / CH_DEPTH_* parameters). The package is not imported by any RTL module — same convention as duck_cache_pkg.sv — because Icarus Verilog 11 / cocotb 2.0 need flat signals rather than packages/structs on module ports. It exists purely as documentation; each ucie_chan instance carries one flit type per instantiation site, with unused fields tied to 0.

Each channel is a single-purpose, single-direction ucie_chan instance (rtl/ucie_chan.sv) modeling:

  1. Credit-based flow control — tx_ready reflects remaining receiver buffer credit (DEPTH), consumed at send and returned when the far side pops. Credit return is idealized as instantaneous (no modeled crossing latency on the return path) — only the forward data path models the die-to-die hop.
  2. Fixed crossing latency — LATENCY cycles (2, in every instantiation) of shift-register delay standing in for SerDes + interposer propagation, applied only forward (tx → rx).
  3. A receive-side buffer (depth DEPTH) so the consumer drains at its own pace.

link_up (from ucie_link_ctrl) gates all activity — while low, tx_ready=0, rx_valid=0, and internal state is held in reset.

rtl/ucie_link_ctrl.sv models the UCIe bring-up sequence at the state-name level (not bit-accurate PHY/eye-training):

RESET → SBINIT → PARAM → MBINIT → MBTRAIN → LINKINIT → ACTIVE

Each pre-ACTIVE state holds for TRAIN_CYCLES cycles (4, in system_top.sv) then advances; ACTIVE is sticky for the design's lifetime — no dynamic retrain or L1/L2 low-power entry is modeled, though the state encoding leaves room for it. One instance per physical link (CPU link, NPU link); link_up gates every ucie_chan on that link.

Handshake Translations

  • instr_load (base → cpu): ld_tx_valid fires on the NoC router's existing cpu_accept single-cycle pulse — same accept semantics as v4. The key change: cpu_ready (the router's own input, gating whether it can accept a new network word) is now wired to the channel's credit (ld_tx_ready) instead of !core_stall. This decouples network ingestion from the CPU's internal MUL/DIV stall cycles — imem's 256-word capacity is the credit pool, the same limit as before, just enforced by the channel instead of by throttling the network. This is the change that exposed BUG-007 — see that writeup for the tradeoff.
  • mem_req / mem_rsp (bidirectional): a small FSM in base_die_top.sv stands in for "riscv_core" from duck_coherency_ctrl's point of view — latches the incoming request, holds cpu_read/write/addr/wdata/byte_en until cpu_stall clears (unmodified coherency controller timing), then ships the response back as a single flit. On the CPU chiplet side, cpu_chiplet_top.sv's bridge FSM turns riscv_core's held mem_read/mem_write/... signals into a single flit send, waits for the response flit, then presents mem_rdata and drops mem_stall on the exact cycle the response arrives — reproducing the L2/coherency timing contract riscv_core already expects, just relocated across a die boundary.
  • npu_disp (base → npu): direct forward of the router's existing npu_valid/npu_instruction hold register. npu_ready (the router's input) now comes from the channel's credit, which only returns once the NPU chiplet's adapter actually pops the flit — i.e. exactly when the NPU is IDLE — reproducing the original noc_router ↔ npu handshake unchanged, plus crossing latency.
  • npu_result (npu → base): the NPU chiplet drains npu's result bus into a local 16-entry FIFO (one slot per accumulator row) as fast as the NPU produces it, fully decoupling the compute pipeline's OUTPUT_ST timing from the UCIe link's latency/credit. A small drain FSM streams the buffered results out at its own pace; on the base die, res_rx_* forwards straight onto the top-level npu_result_valid/index/data/ready pins, unchanged from the pre-split contract.

halt

halt is treated as an auxiliary/debug pin carried alongside — not through — the CPU link's mainband flits, matching how real packages often expose low-rate debug/status signals outside the high-speed lanes. It's computed in cpu_chiplet_top.sv (imem now lives there) and wired straight through system_top unchanged from the top level's point of view. See BUG-007 for a subtlety in how it's gated.

Deliberate Simplifications

  • Zero-latency credit return. Real link bandwidth is symmetric; here only the forward data path costs LATENCY cycles, and credit returns instantly on pop. This keeps the model tractable without changing the functional contract any channel's producer/consumer see.
  • No dynamic retrain / low-power states. ucie_link_ctrl only models bring-up; ACTIVE is sticky.
  • Fixed LATENCY=2 everywhere, not derived from any physical floorplan — a stand-in for "some SerDes + interposer hop," not a timing-closed number.
  • One flit type per channel rather than a multiplexed multi-VC stream sharing one physical channel — sized to this SoC's five known traffic classes instead of a generic crossbar.

Testing

Makefile adds the five new RTL files (ucie_chan.sv, ucie_link_ctrl.sv, cpu_chiplet_top.sv, npu_chiplet_top.sv, base_die_top.sv) ahead of system_top.sv in VERILOG_SOURCES. No testbench changes were needed beyond updating hierarchical paths that reach into riscv_core's register file for checks — dut.cpu_core_inst... became dut.cpu_chiplet_inst.cpu_core_inst... in tb/tb_server_dispatch_unit.py — and raising HALT_DRAIN_CYCLES per BUG-007. Both suites match their pre-split baseline: regression is TESTS=11 PASS=11 FAIL=0, and MOESI integration is 1 PASS / 2 FAIL, unchanged and tracked separately as BUG-006.

Run: docker run --rm -e ASM_TEST=cpu_peak_tests.asm -e MODULE=tb_server_dispatch_unit -v "$(pwd)":/usr/src/server_dispatch riscv_sim:latest (or ./run_sims.command for both suites).