临时关闭Pcie ACS 重定向服务

1. 背景

在PCIe总线中，P2P（Peer-to-Peer）直通是指在PCIe拓扑中两个EndPoint设备之间直接进行数据传输，而不经过RC侧Memory，P2P直通可以提高数据传输的效率和软件设计的灵活性，P2P通信方式如下图所示。

1.1 pcie p2p配置，地址匹配举例

我们以新思的DesignWare ® Cores举例，业界大部分IC设计都是采用新思的pcie ip。

 pcie一般都会有iATU模块，通过配置iATU实现inbound和outbound规则，atu inbound配置有bar匹配和地址匹配，bar匹配只需要一条规则，但是只能配置连续的地址空间，地址匹配可以配置多条规则来匹配不连续的EP侧地址；

EP1 InBound 配置：

base_addr： Bar0 ； target_addr: 0x900000000 (ep的ddr地址)  ; limit_addr长度: 0x10000000

EP2 OutBound 配置: 

base_addr：0x900000000 (ep的ddr地址) ； target_addr: Bar0 ; limit_addr长度: 0x100000000

这种配置方式就可以让ep1和ep2通过Switch完成p2p

outbound 具体配置流程：

Define Outbound Region 1 as:
64 kB I/O region from  0x80000000_d000000 to  0x80000000_d000ffff , to be mapped to  0x00010000 in
the PCIe I/O space.
1. Setup the Region Base and Limit Address Registers.
Write  0xd0000000 to Address { 0x208 } to set the Lower Base Address.
Write  0x80000000 to Address { 0x20C } to set the Upper Base Address.
Write  0xd000ffff to Address { 0x210 } to set the Limit Address.
2. Setup the Target Address Registers.
Write  0x00010000 to Address { 0x214 } to set the Lower Target Address.
Write  0x00000000 to Address { 0x218 } to set the Upper Target Address.
3. Configure the region through the Region Control 1 Register.
Write  0x00000002 to Address { 0x200 } to define the type of the region to be I/O.
4. Enable the region.
Write  0x80000000 to Address { 0x204 } to enable the region.

inbound 地址匹配流程：

Define Inbound Region 0 as, MEM region matching TLPs with addresses in the range  0x00010000 to
0x0005ffff mapped to  0x1000_0000_2000_0000 -  0x1000_0000_2004_ffff in your application
memory space.
1. Setup the Region Base and Limit Address Registers.
Write  0x00010000 to Address { 0x108 } to set the Lower Base Address.
Write  0x00000000 to Address { 0x10C } to set the Upper Base Address.
Write  0x0005ffff to Address { 0x110 } to set the Limit Address
2. Setup the Target Address Registers.
Write  0x20000000 to Address { 0x114 } to set the Lower Target Address.
Write  0x10000000 to Address { 0x118 } to set the Upper Target Address.
3. Configure the region through the Region Control 1 Register.
Write  0x00000000 to Address { 0x100 } to define the type of the region to be MEM.
4. Enable the region.
Write  0x80000000 to Address { 0x104 } to enable the region in address match mode.

2. 查看拓扑

lspci -vt  查看pcie拓扑

3. 设置ACS访问控制

如果说p2p被重定向到了RC侧，RC不会再将数据通过bar空间发送到目的设备去，所以需要关闭p2p的重定向。

pcie spec 关于ACS控制寄存器描述如下：

根据上面的拓扑，关闭p2p重定向命令：

setpci -v -s 68:10.0 ECAP_ACS+6.w=0

4. 永久关闭可以在BISO里关闭：

重启操作系统开机时按del进入BIOS关闭ACS功能，不关VT-d只关闭ACS功能，具体路径：Path: Advanced -> Chipset Configuration -> North Bridge -> IIO Configuration -> Intel VT for Directed I/O (VT-d) -> ACS Control -> Enable / Disable.