NPU v2 — Neural Processing Unit

The NPU v2 (rtl/npu.sv) is a 16×16 weight-stationary fully-parallel MAC array for INT8 GEMV and tiled GEMM. All 256 INT8 multiplications are computed combinationally every cycle; results are registered in a single clock edge.

Peak throughput: 0.512 TOPS @ 1 GHz

What It Does

Operation	Instruction	Result
Matrix-vector multiply	`COMPUTE`	`acc[r] = Σ W[r][c] · A[c]` for all 16 rows
Tile accumulate	`COMPUTE_ACC`	`acc[r] += Σ W[r][c] · A[c]` (adds to existing acc)
Load weight	`LOAD_W`	`W[row][col] = data` (one INT8 element)
Load activation	`LOAD_A`	`A[col] = data` (one INT8 element)
No-op	`NOP`	—

Instruction Encoding (opcode `0x6B`, RISC-V custom-1)

 Bit: 31    26 25    18 17   14 13   10 9   7 6       0
      +-------+---------+-------+-------+-----+---------+
      | unused|  data   |  col  |  row  | cmd | 0x6B    |
      | [31:26]| [25:18]|[17:14]|[13:10]|[9:7]| [6:0]  |
      +-------+---------+-------+-------+-----+---------+

`cmd [9:7]`	Mnemonic	Fields used	Action
`3'b000`	`LOAD_W`	row[13:10], col[17:14], data[25:18]	`W[row][col] ← data`
`3'b001`	`LOAD_A`	col[17:14], data[25:18]	`A[col] ← data`
`3'b010`	`COMPUTE`	—	`acc ← W·A` (overwrite)
`3'b011`	`COMPUTE_ACC`	—	`acc += W·A` (accumulate)
`3'b111`	`NOP`	—	idle

data is INT8 signed — sign-extended automatically by $signed(npu_instruction[25:18]).

Hardware Architecture

The Parallel MAC Array

256 independent multipliers, all running simultaneously every cycle:

// Stage 1: 256 multiplications (INT8 × INT8 → INT16)
logic signed [15:0] mac_prod [0:15][0:15];

generate
    for (gr = 0; gr < 16; gr++) begin : gen_mac_row
        for (gc = 0; gc < 16; gc++) begin : gen_mac_col
            assign mac_prod[gr][gc] =
                $signed(weights[gr][gc]) * $signed(activations[gc]);
        end
    end
endgenerate

Explicit $signed() casts prevent Icarus Verilog from treating the multiply as unsigned when both operands are logic signed arrays.

The Adder Tree

One 16-input adder tree per row sums the INT16 products to INT32:

// Stage 2: 16-input adder tree per row (→ 4-level balanced tree at synthesis)
logic signed [31:0] row_sum [0:15];

generate
    for (rr = 0; rr < 16; rr++) begin : gen_rowsum
        assign row_sum[rr] =
            mac_prod[rr][0] + mac_prod[rr][1] + ... + mac_prod[rr][15];
    end
endgenerate

Synthesisers map this to a 4-level balanced adder tree (log₂(16) = 4). Maximum sum = 16 × 127 × 127 = 258,064 — fits in INT32 with no overflow risk.

Accumulators

16 × INT32 accumulators (acc[0:15]) hold the running results. In COMPUTE they are overwritten; in COMPUTE_ACC the new row_sum is added to the existing value.

State Machine

         npu_valid + cmd=000
   ┌──────────────────────────► LOAD_W_ST ──────────────────────► IDLE
   │                                                           (1 cycle)
   │     npu_valid + cmd=001
   ├──────────────────────────► LOAD_A_ST ──────────────────────► IDLE
   │                                                           (1 cycle)
   │     npu_valid + cmd=010
   ├──────────────────────────► COMPUTE_ST ──────────────────────┐
   │                                    (acc = row_sum, 1 cycle)  │
IDLE                                                              ▼
   │     npu_valid + cmd=011                               OUTPUT_ST
   ├──────────────────────────► COMP_ACC_ST ─────────────────────┤
   │                                 (acc += row_sum, 1 cycle)   │
   │                                                             │
   │     npu_valid + cmd=111                                     │
   └──────────────────────────► IDLE (NOP)      out_cnt==15 + ready
                                                     └──────────► IDLE

State	Cycles	Action
`IDLE`	—	Accept instruction, latch fields, assert `npu_ready`
`LOAD_W_ST`	1	`weights[lat_row][lat_col] ← lat_data`
`LOAD_A_ST`	1	`activations[lat_col] ← lat_data`
`COMPUTE_ST`	1	`acc[r] ← row_sum[r]` for all 16 rows
`COMP_ACC_ST`	1	`acc[r] += row_sum[r]` for all 16 rows
`OUTPUT_ST`	16	Stream `acc[0..15]` one per cycle with backpressure

npu_ready is asserted only in IDLE — the SDU cannot dispatch a new instruction while any other state is active.

Instruction Latch Hazard

The SDU can overwrite npu_instruction in the same clock cycle that the NPU accepts it and transitions out of IDLE. To prevent this:

In the IDLE default case of the datapath always_ff, all instruction fields are captured before the state change:

default: begin   // IDLE
    out_cnt <= 4'd0;
    if (npu_valid) begin
        lat_cmd  <= cmd;
        lat_row  <= row_idx;
        lat_col  <= col_idx;
        lat_data <= data_byte;
    end
end

LOAD_W_ST and LOAD_A_ST then use lat_row, lat_col, lat_data — not the live npu_instruction signals.

Tiled GEMM (K > 16)

The COMPUTE_ACC instruction enables arbitrary-depth GEMM by tiling along the K (inner) dimension.

Example: K=32 dot product (two 16-wide tiles)

# Tile 0 — establish baseline
for r in 0..15:
    for c in 0..15:
        LOAD_W  row=r, col=c, data=W0[r][c]
for c in 0..15:
    LOAD_A  col=c, data=A0[c]
COMPUTE          # acc = W0·A0

# Tile 1 — accumulate
for r in 0..15:
    for c in 0..15:
        LOAD_W  row=r, col=c, data=W1[r][c]
for c in 0..15:
    LOAD_A  col=c, data=A1[c]
COMPUTE_ACC      # acc += W1·A1

# Collect results — 16 INT32 lanes
# acc[r] = Σ_{c=0}^{31} W_full[r][c] · A_full[c]

For K=N×16, chain N-1 COMPUTE_ACC instructions after the first COMPUTE.

Result Bus

After COMPUTE or COMPUTE_ACC, the NPU streams 16 INT32 results one per cycle:

Signal	Direction	Description
`result_valid`	out	High during OUTPUT_ST
`result_index [3:0]`	out	Current lane index (0–15)
`result_data [31:0]`	out	`acc[result_index]` — INT32 signed
`result_ready`	in	Backpressure — hold `result_index` when low

result_index only increments when result_ready is asserted. The NPU stays in OUTPUT_ST until all 16 lanes have been acknowledged.

Processing Element — `rtl/pe.sv`

The architectural atom of the NPU. Every conceptual position in the 16×16 array corresponds to one PE:

              weight (stationary)
                   │
act_in ──────────► [×] ──► (+) ──► sum_out
                           ▲
                        sum_in

always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
        act_out <= 8'sb0;
        sum_out <= 32'sb0;
    end else begin
        act_out <= act_in;                                // pass activation right
        sum_out <= sum_in + (weight_sext * act_in_sext); // accumulate
    end
end

pe.sv exists as a standalone module for future systolic tiling and FPGA DSP-slice mapping. The NPU v2 inlines equivalent combinational logic for maximum Icarus Verilog simulation throughput.

Performance Numbers

Metric	Value
MACs per compute cycle	256 (all 16×16 simultaneously)
Cycles per GEMV (16×16)	256 LOAD_W + 16 LOAD_A + 1 COMPUTE + 16 OUTPUT = 289
Peak MAC throughput	256 MACs/cycle
Peak TOPS @ 1 GHz	0.512 TOPS
Accumulator width	INT32 (no overflow for INT8 inputs)
Max safe accumulation	16 × 127 × 127 = 258,064 per lane
Tiled GEMM depth	Unlimited (COMPUTE_ACC chaining)