NTB Drivers

Non-Transparent Bridge 기능

NTB, Non-Transparent Bridge는 두 대 이상 computer의 분리된 memory system을 같은 PCI Express fabric에 연결하는 bridge chip입니다.

공통 기능은 doorbell register와 memory translation window이며 hardware에 따라 scratchpad와 message register도 제공합니다.

Scratchpad는 양쪽에서 읽고 쓸 수 있는 fixed-address register로 소량 정보를 교환합니다. Message register도 같은 용도지만 다른 peer의 overwrite를 막는 status bit가 있습니다.

Doorbell은 peer interrupt event를 보내고 memory window는 peer memory에 대한 translated read/write access를 제공합니다.

===========
NTB Drivers
===========

NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects
the separate memory systems of two or more computers to the same PCI-Express
fabric. Existing NTB hardware supports a common feature set: doorbell
registers and memory translation windows, as well as non common features like
scratchpad and message registers. Scratchpad registers are read-and-writable
registers that are accessible from either side of the device, so that peers can
exchange a small amount of information at a fixed address. Message registers can
be utilized for the same purpose. Additionally they are provided with
special status bits to make sure the information isn't rewritten by another
peer. Doorbell registers provide a way for peers to send interrupt events.
Memory windows allow translated read and write access to the peer memory.

NTB core와 client driver

NTB core driver는 공통 feature set을 감싼 API를 정의하고 client가 hardware driver가 지원하는 NTB device를 발견하게 합니다.

Client는 NTB API를 사용하는 upper-layer component이고 hardware driver는 특정 vendor와 model의 NTB를 구동합니다.

Client driver는 NTB core에 등록합니다. Hardware나 driver가 삽입·제거되면 probe/remove가 호출됩니다. Linux Device framework를 사용하므로 PCI driver와 유사합니다.

NTB Core Driver (ntb)
=====================

The NTB core driver defines an api wrapping the common feature set, and allows
clients interested in NTB features to discover NTB the devices supported by
hardware drivers.  The term "client" is used here to mean an upper layer
component making use of the NTB api.  The term "driver," or "hardware driver,"
is used here to mean a driver for a specific vendor and model of NTB hardware.

NTB Client Drivers
==================

NTB client drivers should register with the NTB core driver.  After
registering, the client probe and remove functions will be called appropriately
as ntb hardware, or hardware drivers, are inserted and removed.  The
registration uses the Linux Device framework, so it should feel familiar to
anyone who has written a pci driver.

Inbound·outbound memory window

NTB의 주 목적은 둘 이상의 system이 memory를 공유하는 것입니다. Scratchpad/message register는 주로 memory window 초기화 정보를 교환하는 데 사용합니다.

Inbound translation은 local NTB port에 설정합니다. Local device가 DMA-mapped memory를 할당하고 `ntb_mw_set_trans(addr)`로 translation을 설정한 뒤 peer에 알리면 peer가 outbound window를 map해 공유 memory에 접근합니다.

Outbound translation은 peer가 초기화합니다. Local이 memory와 translated address를 peer에 전달하고 peer가 `ntb_peer_mw_set_trans(addr)`로 자신의 NTB config를 설정한 뒤 outbound window를 map합니다.

Portable algorithm은 local이 shared memory를 할당하고 가능한 경우 local window를 초기화한 뒤 translated address와 window index를 peer에 보냅니다. Peer는 가능한 경우 받은 주소로 window를 초기화하고 outbound window를 map합니다.

API 순서는 `ntb_mw_count(pidx)`로 range 수 확인, `ntb_get_align(pidx,midx)`로 alignment·size 제한 확인, physically contiguous memory 할당, `ntb_mw_set_trans()` 설정, scratchpad/message로 base와 window 번호 전달입니다.

Peer는 `ntb_peer_mw_set_trans()`로 받은 주소를 설정합니다. Maximum address 초과나 alignment 오류면 실패할 수 있습니다. `ntb_peer_mw_get_addr(widx)`로 MMIO address를 얻어 map합니다.

Local `ntb_mw_count(pidx)` 값은 반대편 peer에서 대응 port index에 대한 `ntb_peer_mw_count()`와 같아야 합니다.

NTB Typical client driver implementation
----------------------------------------

Primary purpose of NTB is to share some piece of memory between at least two
systems. So the NTB device features like Scratchpad/Message registers are
mainly used to perform the proper memory window initialization. Typically
there are two types of memory window interfaces supported by the NTB API:
inbound translation configured on the local ntb port and outbound translation
configured by the peer, on the peer ntb port. The first type is
depicted on the next figure::

 Inbound translation:

 Memory:              Local NTB Port:      Peer NTB Port:      Peer MMIO:
  ____________
 | dma-mapped |-ntb_mw_set_trans(addr)  |
 | memory     |        _v____________   |   ______________
 | (addr)     |<======| MW xlat addr |<====| MW base addr |<== memory-mapped IO
 |------------|       |--------------|  |  |--------------|

So typical scenario of the first type memory window initialization looks:
1) allocate a memory region, 2) put translated address to NTB config,
3) somehow notify a peer device of performed initialization, 4) peer device
maps corresponding outbound memory window so to have access to the shared
memory region.

The second type of interface, that implies the shared windows being
initialized by a peer device, is depicted on the figure::

 Outbound translation:

 Memory:        Local NTB Port:    Peer NTB Port:      Peer MMIO:
  ____________                      ______________
 | dma-mapped |                |   | MW base addr |<== memory-mapped IO
 | memory     |                |   |--------------|
 | (addr)     |<===================| MW xlat addr |<-ntb_peer_mw_set_trans(addr)
 |------------|                |   |--------------|

Typical scenario of the second type interface initialization would be:
1) allocate a memory region, 2) somehow deliver a translated address to a peer
device, 3) peer puts the translated address to NTB config, 4) peer device maps
outbound memory window so to have access to the shared memory region.

As one can see the described scenarios can be combined in one portable
algorithm.

 Local device:
  1) Allocate memory for a shared window
  2) Initialize memory window by translated address of the allocated region
     (it may fail if local memory window initialization is unsupported)
  3) Send the translated address and memory window index to a peer device

 Peer device:
  1) Initialize memory window with retrieved address of the allocated
     by another device memory region (it may fail if peer memory window
     initialization is unsupported)
  2) Map outbound memory window

In accordance with this scenario, the NTB Memory Window API can be used as
follows:

 Local device:
  1) ntb_mw_count(pidx) - retrieve number of memory ranges, which can
     be allocated for memory windows between local device and peer device
     of port with specified index.
  2) ntb_get_align(pidx, midx) - retrieve parameters restricting the
     shared memory region alignment and size. Then memory can be properly
     allocated.
  3) Allocate physically contiguous memory region in compliance with
     restrictions retrieved in 2).
  4) ntb_mw_set_trans(pidx, midx) - try to set translation address of
     the memory window with specified index for the defined peer device
     (it may fail if local translated address setting is not supported)
  5) Send translated base address (usually together with memory window
     number) to the peer device using, for instance, scratchpad or message
     registers.

 Peer device:
  1) ntb_peer_mw_set_trans(pidx, midx) - try to set received from other
     device (related to pidx) translated address for specified memory
     window. It may fail if retrieved address, for instance, exceeds
     maximum possible address or isn't properly aligned.
  2) ntb_peer_mw_get_addr(widx) - retrieve MMIO address to map the memory
     window so to have an access to the shared memory.

Also it is worth to note, that method ntb_mw_count(pidx) should return the
same value as ntb_peer_mw_count() on the peer with port index - pidx.

Transport와 Netdev client

주요 client는 `ntb_transport`와 `ntb_netdev`입니다. 둘이 함께 NTB를 가로지르는 peer logical link를 만들어 network packet을 교환합니다.

Transport client는 peer link와 message/data용 queue pair를 만들고 Netdev는 queue pair를 사용한 Ethernet device를 생성합니다. Network data는 socket buffer와 transport queue-pair buffer 사이에서 복사됩니다.

Transport는 Netdev 외 용도로도 쓸 수 있지만 문서 시점에는 다른 application이 없습니다.

NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev)
------------------------------------------------------------------

The primary client for NTB is the Transport client, used in tandem with NTB
Netdev.  These drivers function together to create a logical link to the peer,
across the ntb, to exchange packets of network data.  The Transport client
establishes a logical link to the peer, and creates queue pairs to exchange
messages and data.  The NTB Netdev then creates an ethernet device using a
Transport queue pair.  Network data is copied between socket buffers and the
Transport queue pair buffer.  The Transport client may be used for other things
besides Netdev, however no other applications have yet been written.

Ping Pong test client

`ntb_pingpong`은 doorbell과 scratchpad를 시험하고 간단한 NTB client 예제를 제공합니다.

시작 시 link를 enable하고 up을 기다린 뒤 doorbell·scratchpad를 읽고 씁니다. Peer는 각 round마다 한 bit씩 shift되는 doorbell mask로 interrupt해 여러 bit와 vector 동작을 시험합니다.

각 round에서 local scratchpad 0을 읽고 값에 1을 더해 peer scratchpad 0에 쓴 뒤 peer doorbell을 설정합니다.

`unsafe=1`은 알려진 scratchpad/doorbell issue가 있는 hardware 시험을 위험을 감수하고 허용합니다. `delay_ms`는 interrupt 수신과 다음 peer doorbell 사이 delay, `init_db`는 새 round series의 시작 bit입니다. Load 시 `dyndbg=+p`가 권장됩니다.

NTB Ping Pong Test Client (ntb\_pingpong)
-----------------------------------------

The Ping Pong test client serves as a demonstration to exercise the doorbell
and scratchpad registers of NTB hardware, and as an example simple NTB client.
Ping Pong enables the link when started, waits for the NTB link to come up, and
then proceeds to read and write the doorbell scratchpad registers of the NTB.
The peers interrupt each other using a bit mask of doorbell bits, which is
shifted by one in each round, to test the behavior of multiple doorbell bits
and interrupt vectors.  The Ping Pong driver also reads the first local
scratchpad, and writes the value plus one to the first peer scratchpad, each
round before writing the peer doorbell register.

Module Parameters:

* unsafe - Some hardware has known issues with scratchpad and doorbell
        registers.  By default, Ping Pong will not attempt to exercise such
        hardware.  You may override this behavior at your own risk by setting
        unsafe=1.
* delay\_ms - Specify the delay between receiving a doorbell
        interrupt event and setting the peer doorbell register for the next
        round.
* init\_db - Specify the doorbell bits to start new series of rounds.  A new
        series begins once all the doorbell bits have been shifted out of
        range.
* dyndbg - It is suggested to specify dyndbg=+p when loading this module, and
        then to observe debugging output on the console.

NTB Tool debugfs client

`ntb_tool`은 주로 NTB hardware와 driver debugging에 사용하며 module parameter는 없습니다.

Probe된 device마다 `debugfs/ntb_tool/hw` directory를 만들고 doorbell과 mask, peer doorbell/mask, local·peer scratchpad를 읽고 변경합니다.

`db`는 read로 local doorbell을 읽고 `s <bits>` write로 set, `c <bits>`로 clear합니다. `mask`, `peer_db`, `peer_mask`도 같은 형식입니다.

`spad`는 모든 local scratchpad를 읽거나 `<number> <value>` pair series를 써 여러 register를 설정합니다. `peer_spad`는 peer scratchpad에 같은 동작을 합니다.

NTB Tool Test Client (ntb\_tool)
--------------------------------

The Tool test client serves for debugging, primarily, ntb hardware and drivers.
The Tool provides access through debugfs for reading, setting, and clearing the
NTB doorbell, and reading and writing scratchpads.

The Tool does not currently have any module parameters.

Debugfs Files:

* *debugfs*/ntb\_tool/*hw*/
        A directory in debugfs will be created for each
        NTB device probed by the tool.  This directory is shortened to *hw*
        below.
* *hw*/db
        This file is used to read, set, and clear the local doorbell.  Not
        all operations may be supported by all hardware.  To read the doorbell,
        read the file.  To set the doorbell, write `s` followed by the bits to
        set (eg: `echo 's 0x0101' > db`).  To clear the doorbell, write `c`
        followed by the bits to clear.
* *hw*/mask
        This file is used to read, set, and clear the local doorbell mask.
        See *db* for details.
* *hw*/peer\_db
        This file is used to read, set, and clear the peer doorbell.
        See *db* for details.
* *hw*/peer\_mask
        This file is used to read, set, and clear the peer doorbell
        mask.  See *db* for details.
* *hw*/spad
        This file is used to read and write local scratchpads.  To read
        the values of all scratchpads, read the file.  To write values, write a
        series of pairs of scratchpad number and value
        (eg: `echo '4 0x123 7 0xabc' > spad`
        # to set scratchpads `4` and `7` to `0x123` and `0xabc`, respectively).
* *hw*/peer\_spad
        This file is used to read and write peer scratchpads.  See
        *spad* for details.

NTB MSI test client

`ntb_msi_test`는 NTB memory window를 통해 MSI interrupt를 전달하는 MSI library를 시험·debug하며 debugfs로 제어합니다.

Probe된 device마다 `debugfs/ntb_msi_test/hw`를 만듭니다. `port`는 local port number이고 interrupt별 `irq*_occurrences`는 trigger 횟수를 반환합니다.

Peer별 `peer*/port`는 port 번호, `peer*/count`는 trigger 가능한 interrupt 수입니다. `peer*/trigger`에 count보다 작은 interrupt number를 쓰면 해당 peer interrupt를 trigger하고 peer의 occurrence가 증가해야 합니다.

NTB MSI Test Client (ntb\_msi\_test)
------------------------------------

The MSI test client serves to test and debug the MSI library which
allows for passing MSI interrupts across NTB memory windows. The
test client is interacted with through the debugfs filesystem:

* *debugfs*/ntb\_msi\_test/*hw*/
        A directory in debugfs will be created for each
        NTB device probed by the msi test.  This directory is shortened to *hw*
        below.
* *hw*/port
        This file describes the local port number
* *hw*/irq*_occurrences
        One occurrences file exists for each interrupt and, when read,
        returns the number of times the interrupt has been triggered.
* *hw*/peer*/port
        This file describes the port number for each peer
* *hw*/peer*/count
        This file describes the number of interrupts that can be
        triggered on each peer
* *hw*/peer*/trigger
        Writing an interrupt number (any number less than the value
        specified in count) will trigger the interrupt on the
        specified peer. That peer's interrupt's occurrence file
        should be incremented.

Hardware driver 등록

NTB hardware driver는 device를 NTB core에 등록해야 하며 등록 후 client probe/remove가 호출됩니다.

Intel hardware driver `ntb_hw_intel`은 Xeon과 Atom CPU의 NTB를 지원합니다.

NTB Hardware Drivers
====================

NTB hardware drivers should register devices with the NTB core driver.  After
registering, clients probe and remove functions will be called.

NTB Intel Hardware Driver (ntb\_hw\_intel)
------------------------------------------

The Intel hardware driver supports NTB on Xeon and Atom CPUs.

Intel B2B memory-window parameter

`b2b_mw_idx`는 peer NTB를 memory window로 접근할 때 사용할 index입니다. 0 이상은 첫 index부터, 음수는 마지막 index부터 세며 양쪽이 반드시 같은 값을 써야 합니다. 기본은 `-1`입니다.

`b2b_mw_share`는 window가 충분히 클 때 그 절반을 client의 peer address translation에도 허용합니다.

Xeon B2B topology의 `xeon_b2b_usd_bar2_addr64`는 upstream side BAR2 window가 두 NTB 사이 bus에서 사용할 64-bit address입니다.

나머지 USD/DSD BAR4·BAR5 64/32-bit parameter도 BAR2 parameter와 같은 방식으로 각 window address를 지정합니다.

Module Parameters:

* b2b\_mw\_idx
        If the peer ntb is to be accessed via a memory window, then use
        this memory window to access the peer ntb.  A value of zero or positive
        starts from the first mw idx, and a negative value starts from the last
        mw idx.  Both sides MUST set the same value here!  The default value is
        `-1`.
* b2b\_mw\_share
        If the peer ntb is to be accessed via a memory window, and if
        the memory window is large enough, still allow the client to use the
        second half of the memory window for address translation to the peer.
* xeon\_b2b\_usd\_bar2\_addr64
        If using B2B topology on Xeon hardware, use
        this 64 bit address on the bus between the NTB devices for the window
        at BAR2, on the upstream side of the link.
* xeon\_b2b\_usd\_bar4\_addr64 - See *xeon\_b2b\_bar2\_addr64*.
* xeon\_b2b\_usd\_bar4\_addr32 - See *xeon\_b2b\_bar2\_addr64*.
* xeon\_b2b\_usd\_bar5\_addr32 - See *xeon\_b2b\_bar2\_addr64*.
* xeon\_b2b\_dsd\_bar2\_addr64 - See *xeon\_b2b\_bar2\_addr64*.
* xeon\_b2b\_dsd\_bar4\_addr64 - See *xeon\_b2b\_bar2\_addr64*.
* xeon\_b2b\_dsd\_bar4\_addr32 - See *xeon\_b2b\_bar2\_addr64*.
* xeon\_b2b\_dsd\_bar5\_addr32 - See *xeon\_b2b\_bar2\_addr64*.