MCTP transport

MCTP (Management Component Transport Protocol, DMTF DSP0236) is the base transport that carries every management message an OpenSMA controller handles. A host BMC, a peer device, or a downstream GPU all speak to the controller as MCTP endpoints, each identified by an 8-bit Endpoint ID (EID). MCTP itself is medium-independent: the same packet can arrive over SMBus/I2C, I3C, SPI, or USB. OpenSMA’s job is to receive those packets on whatever bus they land, reassemble multi-packet messages, look at the MCTP message type, and hand the body to the right protocol stack (Control, PLDM, SPDM, or the NVIDIA vendor/NSM stacks).

This page is a deep-dive on src/nv/mctp/. See the section landing for the bus drivers that feed it and Management Protocols for the stacks layered on top.

Layering: nv/mctp over corepdk/mctp-cpp

OpenSMA does not re-implement the MCTP wire format. The reusable protocol engine lives in corepdk/modules/mctp-cpp/ (packet layout, composer, control-message handling, validator) with a per-SoC platform layer under corepdk/platforms/mcxn236/mctp-cpp/. The files in src/nv/mctp/ are thin NVIDIA wrappers that pull those corepdk types into the nv::mctp namespace and add the FreeRTOS task, the bus fan-out, and the NVIDIA vendor/NSM message handlers.

The wrapper headers are almost pure type aliases:

  • src/nv/mctp/interface.h:31 aliases Packet to pdk::mctp::app::Packet and src/nv/mctp/interface.h:32 aliases Client to pdk::mctp::platforms::Interface (the enum of physical/logical bindings).
  • src/nv/mctp/composer.h:22 aliases Composer to the corepdk assembler.
  • src/nv/mctp/control.h:23 aliases Control to the platform control class.
  • src/nv/mctp/router.h:28 aliases the routing-table types.
Concern corepdk definition nv/mctp wrapper
Packet / headers corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-packet.h:29 src/nv/mctp/interface.h:31
Message types corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-enums.h:105 via enums.h
Reassembly (composer) corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-composer.h:23 src/nv/mctp/composer.h:22
Control messages corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-control.h:39 src/nv/mctp/control.h:23
Routing table corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-router.h:47 src/nv/mctp/router.h:28
Validator corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-validator.h:26 Driver::Validator alias, src/nv/mctp/driver.h:63

The packet on the wire

The corepdk Packet is a packed struct of three parts — a private (inter-task) header, the 4-byte MCTP transport header, and the message body (corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-packet.h:29):

  • PrivateHeader (corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-packet.h:34) is not on the MCTP wire. It records packet_length and packet_interface so the MCTP task knows which Client a buffered packet belongs to.
  • TransportHeader (corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-common.h:25) is the DSP0236 header: version, dst_eid, src_eid, tag, and the som/eom/pkt_seq fields that drive reassembly.
  • The body is a TransmitUnit-byte payload; the mcxn236 platform sets TransmitUnit = 64 (corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-packet.h:25), so one MCTP packet carries 64 bytes and larger messages span multiple packets.

The message type in the first body byte selects the stack. The corepdk MsgType enum (corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-enums.h:105) defines Control = 0x00, Pldm = 0x01, Spdm = 0x05, VendorPci = 0x7e, and VendorIani = 0x7f.

The MCTP task

src/nv/mctp/task.h:26 defines nv::mctp::Task (a FreeRTOS ipc::Task). Task::make() (src/nv/mctp/task.cpp:32) allocates the stack, reads the board UUID from flash, and registers the enumeration timers. Task::entrypoint() (src/nv/mctp/task.cpp:75) then constructs the Driver and calls its event loop:

Driver drv(task, task.uuid);           // src/nv/mctp/task.cpp:83
nv::bootloader::Driver::set_task_booted(nv::ipc::BootedEventBits::Mctp);
drv.main();                            // src/nv/mctp/task.cpp:85

The Driver class (src/nv/mctp/driver.h:50) owns the protocol objects — _control, _vendor, _nsm, _validator, _composer — and the set of IPC queues it services. Its constructor (src/nv/mctp/driver.cpp:64) wires up five queues by ID: MctpDataRequest (raw bus packets), MctpPldmRequest, MctpSpdmRequest, MctpCmd (the command mailbox), and RoutingTable (src/nv/mctp/driver.cpp:69).

Ingress: how a packet reaches the task

Bus drivers do not call into MCTP directly. When an I2C/I3C/SPI/USB driver has an MCTP packet, it hands it to the MCTP task through a pair of static helpers on Driver. mctp_send() (src/nv/mctp/driver.cpp:1063) pushes the raw packet onto MctpDataRequest and then posts a one-word command — whose value is the source Client enum — onto MctpCmd:

const Command Cmd{.cmd = static_cast<uint16_t>(client)};  // src/nv/mctp/driver.cpp:1072
// ... cmd_queue.send(cmd_item, 100ms);

The command queue is the single serialization point. PLDM and SPDM use their own request queues via the mctp_send(item, id, client) overload (src/nv/mctp/driver.cpp:1015), reached through the convenience wrappers mctp_send_from_pldm() (src/nv/mctp/driver.cpp:1083) and mctp_send_from_spdm() (src/nv/mctp/driver.cpp:1088). Internal events (enumeration ticks, endpoint status changes, watchdog) go through mctp_send_cmd() (src/nv/mctp/driver.cpp:1034), which posts a CmdCode (src/nv/mctp/driver.h:68) instead of a Client.

The command loop

Driver::main() (src/nv/mctp/driver.cpp:141) blocks on MctpCmd (src/nv/mctp/driver.cpp:157) and switches on the command word (src/nv/mctp/driver.cpp:160). Because the low command values are the Client enum, a bus-sourced command falls into the on_receive(client) arm (src/nv/mctp/driver.cpp:177); higher CmdCode values dispatch to the enumeration, endpoint-status, and event handlers. There is also a weak mctp_customize_1() hook (src/nv/mctp/driver.cpp:58) dispatched by CmdCode::MctpCustomize1 so a project can run its own work in the MCTP task context without patching the loop.

Receive, reassemble, dispatch

Driver::on_receive(Client) (src/nv/mctp/driver.cpp:269) is the heart of the transport:

  1. Pop the matching packet from MctpDataRequest into _rx_buf.
  2. Validate it with the corepdk validator (src/nv/mctp/driver.cpp:283) — the validator (corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-validator.h:30) checks the tag/sequence against the routing table and drops malformed frames.
  3. Reassemble if needed: when som && eom are not both set the packet is part of a multi-packet message, so _composer.recv_packet() accumulates it into _multi_pkt_buf (src/nv/mctp/driver.cpp:293). The composer (corepdk/modules/mctp-cpp/src/app/pdk-mctp-app-composer.h:37) returns true only on the last fragment.
  4. Dispatch on the message type (src/nv/mctp/driver.cpp:326):
MsgType Handler Source line
Control _control.process(rx, tx) src/nv/mctp/driver.cpp:332
Pldm forward(Client::Pldm, rx) src/nv/mctp/driver.cpp:340
Spdm forward(Client::Spdm, rx) src/nv/mctp/driver.cpp:346
VendorPci _nsm.process(rx, tx) src/nv/mctp/driver.cpp:353
VendorIani _vendor.process(rx, tx) src/nv/mctp/driver.cpp:363

Control messages are answered in-task by the platform control class (corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-control.h:43); Pldm/Spdm bodies are forwarded to their own tasks; VendorPci bodies go to the NSM stack (src/nv/mctp/nsm.h:1391, process at src/nv/mctp/nsm.h:1395) and VendorIani bodies to the NVIDIA vendor stack (src/nv/mctp/vendor.h:36, process at src/nv/mctp/vendor.h:40).

Egress

Any produced response is emitted through on_forward_message() (src/nv/mctp/driver.cpp:443), which uses the composer to re-fragment messages larger than one packet and stamps the destination EID before calling forward(). Driver::forward() (src/nv/mctp/driver.cpp:381) is the bus fan-out — it maps the destination Client back to a concrete bus task: nv::i2c::Task::tx (src/nv/mctp/driver.cpp:389), nv::i3c::Task::tx (src/nv/mctp/driver.cpp:403), nv::spi::Task::tx, or usb::Task::usb_tx (src/nv/mctp/driver.cpp:410), plus the PLDM/SPDM task queues.

flowchart TD
  BUS["Bus ISR / task (I2C, I3C, SPI, USB)"] -->|mctp_send| DQ["MctpDataRequest queue"]
  BUS -->|command = Client id| CQ["MctpCmd queue"]
  CQ --> MAIN["Driver::main command loop"]
  MAIN -->|Client id| RECV["on_receive"]
  DQ -.raw packet.-> RECV
  RECV --> VAL["validator.validate"]
  VAL --> ASM["composer.recv_packet reassemble"]
  ASM --> SW{"msg_type"}
  SW -->|Control| CTL["control.process - reply in task"]
  SW -->|PLDM| PLDM["forward to PLDM task"]
  SW -->|SPDM| SPDM["forward to SPDM task"]
  SW -->|VendorPci| NSM["nsm.process"]
  SW -->|VendorIani| VEN["vendor.process"]
  CTL --> FWD["on_forward_message / forward"]
  NSM --> FWD
  VEN --> FWD
  FWD --> OUT["i2c/i3c/spi Task::tx or usb usb_tx"]

Endpoints, EIDs, and enumeration

An OpenSMA controller is usually an MCTP bridge: it owns an upstream link to the BMC and enumerates a set of downstream endpoints (GPUs, peer devices) so it can assign them EIDs and route between them. The bindings are the Client/Interface enum (corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-enums.h:24): UsI2c/UsUsb upstream, DsI2c0..7 and DsI3c0/1 and Spi0..2 downstream. src/nv/mctp/interface.h:38 maps each Client to an event bit.

State lives in the routing table. ShardRoutingTable (corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-router.h:30) tracks per-endpoint is_need_enumerate / is_enumerated / assigned_eid, while RoutingTable (corepdk/platforms/mcxn236/mctp-cpp/src/pdk-mctp-platforms-router.h:47) holds the controller’s own EC/MC EID maps and UUID. Driver::init() (src/nv/mctp/driver.cpp:105) marks every downstream entry as needing enumeration and brings up the I3C buses.

Enumeration is timer-driven. Task::make() registers MctpEnumerate and MctpEnumerateStart timers (src/nv/mctp/task.cpp:49); the loop’s CmdCode::Enumerate arm runs Driver::on_enumerate() (src/nv/mctp/driver.cpp:484), which walks the downstream map and issues Set-Endpoint-ID control messages via _control.on_gen_set_endpoint_id() (src/nv/mctp/driver.cpp:516). The period constant is Driver::EnumeratePeriod = 3500ms (src/nv/mctp/driver.h:57).

Inter-core IPC (core0 ↔ core1)

On dual-core SoCs the MCTP task runs on core0, but some traffic (USB in the NCSI build) is handled by a bare-metal stack on core1. The nv::ipc subsystem provides the Core-to-Core (C2C) transport that lets a queue send() or event on one core surface on the other.

The abstract ipc::Queue (src/nv/ipc/queue.h:35), accessed by ID through Queue::make() (src/nv/ipc/queue.h:59), is the same API the MCTP task uses locally; when the destination lives on the other core, the send is marshalled by the IPC task. nv::ipc::task::Task (src/nv/ipc/ipc_task.h:38) runs a loop (src/nv/ipc/ipc_task.cpp:95) that reads the shared StreamBuffer (src/nv/ipc/ipc_task.cpp:107) and, in the other direction, serializes queue and event requests into a fixed wire format (handle_queue_data, src/nv/ipc/ipc_task.cpp:252; handle_event_data, src/nv/ipc/ipc_task.cpp:295) before triggering the peer-core interrupt (src/nv/ipc/ipc_task.cpp:283).

Because the two cores may be built with different compilers, the request is never sent as a raw std::variant. Instead it is packed into the explicit QueueRequestWire / EventRequestWire structs (src/nv/ipc/wire_format.h:29 and :41), each a fixed 16 bytes. The shared ring buffer itself is described by C2CStreamBufferCtrl (src/nv/ipc/c2c_stream_buffer.h:37), which mirrors the FreeRTOS StaticStreamBuffer_t layout so the bare-metal core can read/write it without FreeRTOS headers. The low-level driver API is nv::ipc::task::Driver (src/nv/ipc/driver.h:24), with write() (src/nv/ipc/driver.h:31) and read() (src/nv/ipc/driver.h:33).

Interpretation: this two-queue-plus-wire-format design means the MCTP task’s code is identical whether a downstream client is local or lives on the other core — the Client-to-queue mapping and the IPC marshalling hide the core boundary.

Mailbox (mainbox)

Separately from the queue-based IPC, src/nv/mainbox/mailbox.h:23 defines a tiny fixed-address shared-memory “mailbox” for small, latched values that must survive across contexts. It reserves a 0x100-byte region at 0x20002300 (src/nv/mainbox/mailbox.h:25 and :27) partitioned by MainBoxMemoryType (src/nv/mainbox/mailbox.h:29) — e.g. ELS self-test result, fault/WDT records, board serial number, USB-port-reset flag, and a peer thermal-warning word. The MainBoxMemoryDescs table (src/nv/mainbox/mailbox.h:47) gives each type an offset and size, and the inline write_mailbox() / read_mailbox() helpers (src/nv/mainbox/mailbox.h:61 and :73) memcpy to/from the fixed address. Unlike the MCTP queues this is not a transport — it is a handful of memory-mapped registers used for cross-boot and cross-core signalling.

Where to go next

  • Bus drivers — the I2C/I3C/SPI/USB/SSIF drivers that call mctp_send() and receive from forward().
  • Management Protocols — PLDM, SPDM, and the NSM vendor stack dispatched from on_receive.
  • Architecture — the dual-core build that the IPC layer bridges.

Built with Just the Docs. Source-traced against NVIDIA/OpenSMA at 9b183d3 (v02.0020.0000). Methodology under appendices/trace-methodology.

This site uses Just the Docs, a documentation theme for Jekyll.