• AMD GPU 内核驱动分析(三)-dma-fence 同步工作模型

    在Linux Kernel 的AMDGPU驱动实现中，dma-fence扮演着重要角色，AMDGPU的Render/解码操作可能涉及到多个方面同时引用buffer的情况，以渲染/视频解码场景为例，应用将渲染/解码命令写入和GPU共享的BUFFER之后，需要将任务提交给GPU的运行队列等待执行，这个时候，应用不仅要监听buffer中任务的执行情况，还要保证在任务执行完毕之前，不能再去修改BUFFER中的数据。而AMDGPU也需要排它性地执行BUFFER命令。在GPU执行结束，应用希望及时得到执行完的信息，以便回收BUFFER重新利用，通知一般由绑定到某个BUFFER上的中断完成。这些操作，如果使用经典的共享BUFFER+锁的方式进行保护和同步，不但效率十分低下，而且各类内核机制杂糅在一起，缺乏一个统一的管理框架，使代码难以维护。 dma-fence则提供了一套简单便捷的机框架，将原子操作，休眠唤醒，同步/异步事件通知等功能嵌入到各种类型的BUFFER管理框架中，将各类机制有机的结合在一起，减少了用户态空间的忙等，使buffer使用更加智能高效。

    以AMDGPU解码视频为例，利用dma-fence ring buffer隔离了应用解码请求和解码任务任务本身，提交任务和解码完成通知均通过BUFFER绑定的dma-fence进行交互：

    为了便于分析，我把5.4内核中AMDGPU这部分的实现提取出来，写了一个可以独立加载执行的内核模块demo，类似于CMODEL，任务提交上下文和解码完成通知上下文在AMDGPU驱动中分别用内核线程和中断实现，在demo中则全部用内核线程实现.

    工作模型如下图所示：

    fence array有256个slot,每个位置代表一个fence.构成一个RING BUFFER。sync_seq为写指针序号，sync_seq只增不减，sync_seq mod 256为在array中的index，相当于RING BUFFER的写指针。

    last_seq代表当前已经消耗掉结束位置（已完成处理的和fence绑定的buffer)的序号，同sync_seq一样，它也是单调递增的自然序列，last_seq mod 256 为在array中的index，相当于ring buffer的读指针。

    而fence_seq则表示写位置的一个抽样，这是一个在AMDGPU中的hardware ring是所有进程共享的，每个进程随时可能会提交新的渲染任务，所以sync_seq时刻在更新，不适合处理。fence_seq则是某个时间点的快照，在last_seq和fence_seq之间的buffer(fence)，都会在下一批中一并得到signal.

    当提交过快，但是消耗较慢时，写方追上读方，此时需要进行生产-消费之间的同步，读端将会反压到写端，写端调用dma_fence_wait进行同步：

    当消费方完成下一个任务后，释放当前slot，并且调用dma_fence_signal唤醒写线程，写线程受到信号后，继续执行，向fence array中填充新的任务：

    dma fence 的free

    拔出萝卜带出泥，通过job fence的free触发schedule fence和finish fence的释放，释放是通过引用计数归零触发的回调，在回调用中利用RCU异步释放FENCE占用的空间。

    以上就是fence array的使用过程中的几个corner case.

    AMDGPU中gpu_sched的TRACE Point

    Linux 内核调试框架提供了GPU调度器的trace event功能，可以在不重新编译模块的情况下，追踪GPU的执行情况，路径如下：

    /sys/kernel/debug/tracing/events/gpu_scheduler

    trace结果：

    1. root@zlcao-A520MS:/sys/kernel/debug/tracing/events/gpu_scheduler# echo 1 > drm_run_job/enable 
    2. root@zlcao-A520MS:/sys/kernel/debug/tracing/events/gpu_scheduler# echo 1 > drm_sched_job/enable 
    3. root@zlcao-A520MS:/sys/kernel/debug/tracing/events/gpu_scheduler# echo 1 >
    4. drm_run_job/            drm_sched_job/          drm_sched_job_wait_dep/ drm_sched_process_job/  enable                  filter                  
    5. root@zlcao-A520MS:/sys/kernel/debug/tracing/events/gpu_scheduler# echo 1 > enable
    6. root@zlcao-A520MS:/sys/kernel/debug/tracing/events/gpu_scheduler# cat ../../trace

    fence 流量控制

    ring结构体中包含一个名子为num_hw_submission的字段，fence array的空间是就是根据其两倍大小来分配的。

    这个变量最早由amdgpu.ko的amdgpu_sched_hw_submission参数传入，AMDGPU驱动会根据RINT的类型再做进一步的调整。

    之后在创建调度线程时传递给sched->hw_submission_limit变量，.如果当前RING上发射出去的正在执行的job packet的数量小于sched->hw_submission_limit时，提交JOB的上向文将会对RING调度内核线程执行唤醒操作。通过这种方式对RING中的执行情况进行流量控制。

    只有KIQ Ring的submission 数目最大，达到了256,原因前图注释已经说明，可能是因为在其上运行的COMMAND数量大。

    ubuntu图形界面常态工作时，AMDGPU ring的负载情况：

    在运行UBUNTU图形终端时，AMDGPU主要有三个RING在执行任务，包含gfx_low/high, sdma，gfx ring负责执行具体的渲染命令，而sdma则负责数据拷贝。

    KFD Ring和GFX Ring的处理方式区别

    KFD ring是支持compute的ring类型，它是用户台提交command packet的，所以不依赖于内核态gpu scheduler.而gfx ring则需要。

    1. > It seems the HSA KFD Driver are live in the DRM frameworkm, which 
    2. > is like the AMD Graphc drivers. But it seems the Graphic Drivers are 
    3. > free to KFD, because the GPU Graphic function are running well if 
    4. > CONFIG_HSA_AMD macro disable  the KFD functions.
    5. >
    6. > but did the KFD driver needs the Drm drivers? *without the DRM 
    7. > Framework, did the KFD driver work normally? *
    8. > and another question,*did the gpu_sched module is share for Graphici 
    9. > driver and KFD? or* it just use for Graphic Drivers.
    10. >
    11. > i am not familar the AMD GPU driver but i am very intrested on it when 
    12. > i found it, thanks very much for your kindly help on my misunderstading.
    13. >
    14. The AMDGPU driver is a unified driver that supports both graphics and 
    15. compute functionality on a wide range of AMD GPUs. You can think of KFD 
    16. as an extension to AMDGPU to support ROCm user mode (HIP, OpenMP, 
    17. PyTorch, Tensorflow, etc.). ROCm user mode still depends on some of the 
    18. DRM functionality. So KFD cannot exist as a standalone driver without 
    19. AMDGPU and DRM.
    20.  
    21. KFD does not use the gpu_sched scheduler. It is only used by the 
    22. graphics driver. Instead it relies on user-mode-queues, where user mode 
    23. places command directly into memory-mapped queues that are executed by 
    24. the firmware on the GPU. We do not use a driver ioctl API for command 
    25. submission, so we don't use the gpu_sched scheduler. Instead the user 
    26. mode compute queues are scheduled by the GPU firmware.

    附模块code：

    1. #include <linux/kernel.h>
    2. #include <linux/init.h>
    3. #include <linux/types.h>
    4. #include <linux/spinlock.h>
    5. #include <linux/module.h>
    6. #include <linux/fs.h>
    7. #include <linux/errno.h>
    8. #include <linux/mm.h>
    9. #include <linux/sched/signal.h>
    10. #include <linux/dma-fence.h>
    11. #include <linux/slab.h>
    12. #include <linux/delay.h>
    13. #include <linux/kthread.h>
    14. #include <linux/sched.h>
    15. #include <uapi/linux/sched/types.h>
    16.  
    17. #define assert(expr)   \
    18.         if (!(expr)) { \
    19.                 printk( "Assertion failed! %s,%s,%s,line=%d\n",\
    20.                                 #expr,__FILE__,__func__,__LINE__); \
    21.                 BUG(); \
    22.         }
    23.  
    24. #define num_hw_submission               128
    25. struct fence_driver {
    26. 	uint64_t                        gpu_addr;
    27. 	volatile uint32_t               *cpu_addr;
    28. 	uint32_t                        sync_seq;
    29. 	atomic_t                        last_seq;
    30. 	bool                            initialized;
    31. 	bool                            initialized_emit;
    32. 	bool                            initialized_recv;
    33. 	unsigned                        num_fences_mask;
    34. 	spinlock_t                      lock;
    35. 	struct dma_fence                **fences;
    36. 	struct mutex                    mutex;
    37. 	struct dma_fence                *last_fence;
    38. 	struct timer_list               timer;
    39. 	struct timer_list               work_timer;
    40. 	wait_queue_head_t               job_scheduled;
    41. };
    42.  
    43. struct fence_set {
    44. 	struct dma_fence                scheduled;
    45. 	struct dma_fence                finished;
    46. 	spinlock_t                      lock;
    47. };
    48.  
    49. struct job_fence {
    50. 	struct dma_fence                job;
    51. 	void                            *data;
    52. };
    53.  
    54. static uint32_t fence_seq;
    55. static struct fence_driver *ring;
    56. static struct task_struct *fence_emit_task;
    57. static struct task_struct *fence_recv_task;
    58. static struct kmem_cache *job_fence_slab;
    59. static struct kmem_cache *sched_fence_slab;
    60.  
    61. static const char *dma_fence_get_name(struct dma_fence *fence)
    62. {
    63. 	return "dma-fence-drv";
    64. }
    65.  
    66. static bool dma_fence_enable_signal(struct dma_fence *fence)
    67. {
    68. 	if (!timer_pending(&ring->work_timer)) {
    69. 		mod_timer(&ring->work_timer, jiffies + HZ / 10);
    70. 	}
    71.  
    72. 	printk("%s line %d, signal fenceno %lld.\n", __func__, __LINE__, fence->seqno);
    73.  
    74. 	return true;
    75. }
    76.  
    77. void fencedrv_fence_free(struct rcu_head *rcu)
    78. {
    79. 	struct fence_set *fs;
    80. 	struct dma_fence *f = container_of(rcu, struct dma_fence, rcu);
    81. 	struct job_fence *jb = container_of(f, struct job_fence, job);
    82.  
    83. 	assert(jb->data != NULL);
    84.  
    85. 	fs = (struct fence_set *)jb->data;
    86.  
    87. 	// the dma_fence_get must be symmentry with dma_fence_put during all the alive.
    88. 	assert((kref_read(&fs->scheduled.refcount) == 1));
    89. 	assert((kref_read(&fs->finished.refcount) == 1));
    90.  
    91. 	//dma_fence_put(&fs->scheduled);
    92. 	dma_fence_put(&fs->finished);
    93. 	kmem_cache_free(job_fence_slab, jb);
    94.  
    95. 	// dump_stack();
    96. }
    97.  
    98. static void fencedrv_dma_fence_release(struct dma_fence *fence)
    99. {
    100. 	// typically usage for dma fence release by rcu.
    101. 	call_rcu(&fence->rcu, fencedrv_fence_free);
    102. }
    103.  
    104. static const struct dma_fence_ops fence_ops = {
    105. 	.get_driver_name = dma_fence_get_name,
    106. 	.get_timeline_name = dma_fence_get_name,
    107. 	.enable_signaling = dma_fence_enable_signal,
    108. 	.release = fencedrv_dma_fence_release,
    109. };
    110.  
    111. static int32_t fencedrv_get_ring_avail(void)
    112. {
    113. 	uint32_t read_seq, write_seq;
    114.  
    115. 	do {
    116. 		read_seq = atomic_read(&ring->last_seq);
    117. 		write_seq = ring->sync_seq;
    118. 	} while (atomic_read(&ring->last_seq) != read_seq);
    119.  
    120. 	read_seq &= ring->num_fences_mask;
    121. 	write_seq &= ring->num_fences_mask;
    122.  
    123. 	pr_err("%s line %d, read_seq %d, write_seq %d.\n",
    124. 	       __func__, __LINE__, read_seq, write_seq);
    125.  
    126. 	if (read_seq <= write_seq) {
    127. 		return write_seq - read_seq;
    128. 	} else {
    129. 		return write_seq + num_hw_submission * 2 - read_seq;
    130. 	}
    131. }
    132.  
    133. static const char *dma_fence_get_name_scheduled(struct dma_fence *fence)
    134. {
    135. 	return "dma-fence-scheduled";
    136. }
    137.  
    138. static const char *dma_fence_get_name_finished(struct dma_fence *fence)
    139. {
    140. 	return "dma-fence-finished";
    141. }
    142.  
    143. static void sched_fence_free(struct rcu_head *head);
    144. static void fencedrv_dma_fence_release_scheduled(struct dma_fence *fence)
    145. {
    146. 	struct fence_set *fs = container_of(fence, struct fence_set, scheduled);
    147.  
    148. 	// typically usage for dma fence release by rcu.
    149. 	call_rcu(&fs->finished.rcu, sched_fence_free);
    150. }
    151.  
    152. static void fencedrv_dma_fence_release_finished(struct dma_fence *fence)
    153. {
    154. 	struct fence_set *fs = container_of(fence, struct fence_set, finished);
    155. 	dma_fence_put(&fs->scheduled);
    156. 	//while (1);
    157. }
    158.  
    159.  
    160. static const struct dma_fence_ops fence_scheduled_ops = {
    161. 	.get_driver_name = dma_fence_get_name_scheduled,
    162. 	.get_timeline_name = dma_fence_get_name_scheduled,
    163. 	.release = fencedrv_dma_fence_release_scheduled,
    164. };
    165.  
    166. static const struct dma_fence_ops fence_finished_ops = {
    167. 	.get_driver_name = dma_fence_get_name_finished,
    168. 	.get_timeline_name = dma_fence_get_name_finished,
    169. 	.release = fencedrv_dma_fence_release_finished,
    170. };
    171.  
    172. static struct fence_set *to_sched_fence(struct dma_fence *f)
    173. {
    174. 	if (f->ops == &fence_scheduled_ops) {
    175. 		return container_of(f, struct fence_set, scheduled);
    176. 	}
    177.  
    178. 	if (f->ops == &fence_finished_ops) {
    179. 		return container_of(f, struct fence_set, finished);
    180. 	}
    181.  
    182. 	return NULL;
    183. }
    184.  
    185. static void sched_fence_free(struct rcu_head *head)
    186. {
    187. 	struct dma_fence *f = container_of(head, struct dma_fence, rcu);
    188. 	struct fence_set *fs = to_sched_fence(f);
    189. 	if (fs == NULL)
    190. 		return;
    191.  
    192. 	assert(f == &fs->finished);
    193.  
    194. 	kmem_cache_free(sched_fence_slab, fs);
    195. 	dump_stack();
    196. }
    197.  
    198. static struct fence_set *init_fence_set(void)
    199. {
    200. 	struct fence_set *fs = kmem_cache_alloc(sched_fence_slab, GFP_KERNEL);
    201. 	if (fs == NULL) {
    202. 		pr_err("%s line %d, alloc fence set from fence set slab failure.\n",
    203. 		       __func__, __LINE__);
    204. 		return NULL;
    205. 	}
    206.  
    207. 	spin_lock_init(&fs->lock);
    208.  
    209. 	dma_fence_init(&fs->scheduled, &fence_scheduled_ops, &fs->lock, 0, 0);
    210. 	dma_fence_init(&fs->finished, &fence_finished_ops, &fs->lock, 0, 0);
    211.  
    212. 	return fs;
    213. }
    214.  
    215. // ref amdgpu_fence_process
    216. static int fence_recv_task_thread(void *data)
    217. {
    218. 	struct sched_param sparam = {.sched_priority = 1};
    219. 	sched_setscheduler(current, SCHED_FIFO, &sparam);
    220.  
    221. 	//mutex_lock(&ring->mutex);
    222. 	while (ring->initialized == false) {
    223. 		set_current_state(TASK_UNINTERRUPTIBLE);
    224. 		if (ring->initialized == true) {
    225. 			break;
    226. 		}
    227. 		//mutex_unlock(&ring->mutex);
    228. 		schedule();
    229. 		//mutex_lock(&ring->mutex);
    230. 	}
    231.  
    232. 	set_current_state(TASK_RUNNING);
    233. 	//mutex_unlock(&ring->mutex);
    234.  
    235. 	while (!kthread_should_stop() && ring->initialized_recv == true) {
    236. 		uint32_t seqno_next = 0;
    237. 		uint32_t seq, last_seq;
    238. 		int r;
    239.  
    240. 		do {
    241. 			// last_seq is the read pointer of fence ring buffer.
    242. 			last_seq = atomic_read(&ring->last_seq);
    243. 			seq = *ring->cpu_addr;
    244.  
    245. 			if (kthread_should_stop())
    246. 				return 0;
    247. 		} while (atomic_cmpxchg(&ring->last_seq, last_seq, seq) != last_seq);
    248.  
    249. 		if (del_timer(&ring->work_timer) &&
    250. 		    seq != ring->sync_seq) {
    251. 			mod_timer(&ring->work_timer, jiffies + HZ / 10);
    252. 		}
    253.  
    254. 		//printk("%s line %d, last_seq %d, seq %d, sync_seq %d.\n", __func__, __LINE__, last_seq, seq, ring->sync_seq);
    255.  
    256. 		if (unlikely(seq == last_seq)) {
    257. 			msleep(10);
    258. 			continue;
    259. 		}
    260.  
    261. 		assert(seq > last_seq);
    262.  
    263. 		last_seq &= ring->num_fences_mask;
    264. 		seq &= ring->num_fences_mask;
    265.  
    266. 		//printk("%s line %d, last_seq %d, seq %d, sync_seq %d.\n", __func__, __LINE__, last_seq, seq, ring->sync_seq);
    267. 		do {
    268. 			struct dma_fence *fence, **ptr;
    269.  
    270. 			++last_seq;
    271. 			last_seq &= ring->num_fences_mask;
    272. 			ptr = &ring->fences[last_seq];
    273. 			fence = rcu_dereference_protected(*ptr, 1);
    274.  
    275. 			RCU_INIT_POINTER(*ptr, NULL);
    276. 			if (!fence) {
    277. 				continue;
    278. 			}
    279.  
    280. 			if (seqno_next == 0 || seqno_next == fence->seqno) {
    281. 				seqno_next = fence->seqno + 1;
    282. 			} else { /*if (seqno_next != 0 && seqno_next != fence->seqno)*/
    283. 				pr_err("%s line %d, seqno is not continue, exptect %d, actual %lld.\n",
    284. 				       __func__, __LINE__, seqno_next, fence->seqno);
    285. 			}
    286.  
    287. 			printk("%s line %d, last_seq/slot %d, seq %d, signal %lld.\n",
    288. 			       __func__, __LINE__, last_seq, seq, fence->seqno);
    289.  
    290. 			if (list_empty(&fence->cb_list)) {
    291. 				printk("%s line %d, fence cb list is empty.\n",
    292. 				       __func__, __LINE__);
    293. 			} else {
    294. 				printk("%s line %d, fence cb list is not empty.\n",
    295. 				       __func__, __LINE__);
    296. 			}
    297.  
    298. 			r = dma_fence_signal(fence);
    299. 			if (kthread_should_stop()) {
    300. 				dma_fence_put(fence);
    301. 				return 0;
    302. 			}
    303.  
    304. 			if (r) {
    305. 				pr_err("%s line %d, fence already signaled.\n",
    306. 				       __func__, __LINE__);
    307. 				continue;
    308. 				//BUG();
    309. 			}
    310.  
    311. 			dma_fence_put(fence);
    312. 		} while (last_seq != seq);
    313.  
    314. 		wake_up(&ring->job_scheduled);
    315. 	}
    316.  
    317. 	set_current_state(TASK_RUNNING);
    318.  
    319. 	return 0;
    320. }
    321.  
    322. // ref amdgpu_fence_emit.
    323. static int fence_emit_task_thread(void *data)
    324. {
    325. 	int r;
    326. 	uint64_t oldwaitseqno = 0;
    327. 	struct sched_param sparam = {.sched_priority = 1};
    328.  
    329. 	sched_setscheduler(current, SCHED_FIFO, &sparam);
    330.  
    331. 	//mutex_lock(&ring->mutex);
    332. 	while (ring->initialized == false) {
    333. 		set_current_state(TASK_UNINTERRUPTIBLE);
    334. 		if (ring->initialized == true) {
    335. 			break;
    336. 		}
    337.  
    338. 		//mutex_unlock(&ring->mutex);
    339. 		schedule();
    340. 		//mutex_lock(&ring->mutex);
    341. 	}
    342.  
    343. 	set_current_state(TASK_RUNNING);
    344. 	//mutex_unlock(&ring->mutex);
    345.  
    346. 	while (!kthread_should_stop() && ring->initialized_emit == true) {
    347. #if 0
    348. 		msleep(1000);
    349. 		printk("%s line %d.\n", __func__, __LINE__);
    350. #else
    351. 		struct dma_fence __rcu **ptr;
    352. 		struct job_fence *fence;
    353. 		uint32_t seq;
    354. 		struct fence_set *fs = init_fence_set();
    355.  
    356. 		fence = kmem_cache_alloc(job_fence_slab, GFP_KERNEL);
    357. 		if (fence == NULL) {
    358. 			pr_err("%s line %d, alloc fence from fence slab failure.\n",
    359. 			       __func__, __LINE__);
    360. 			return -1;
    361. 		}
    362.  
    363. 		// ring->sync_seq is fence ring write pointer.
    364. 		seq = ++ring->sync_seq;
    365. 		dma_fence_init(&fence->job, &fence_ops, &ring->lock, 0, seq);
    366. 		fence->data = fs;
    367.  
    368. 		ptr = &ring->fences[seq & ring->num_fences_mask];
    369.  
    370. 		//printk("%s line %d, seq = %d.\n", __func__, __LINE__, seq);
    371.  
    372. 		if (kthread_should_stop()) {
    373. 			// will call fence_ops.release directly to free the fence.
    374. 			dma_fence_put(&fence->job);
    375. 			continue;
    376. 		}
    377.  
    378. 		if (unlikely(rcu_dereference_protected(*ptr, 1))) {
    379. 			struct dma_fence *old;
    380. 			int diff;
    381.  
    382. 			rcu_read_lock();
    383. 			old = dma_fence_get_rcu_safe(ptr);
    384. 			rcu_read_unlock();
    385.  
    386. 			if (old) {
    387. 				mutex_lock(&ring->mutex);
    388. 				//dma_fence_get(old);
    389. 				ring->last_fence = old;
    390. 				mutex_unlock(&ring->mutex);
    391.  
    392. 				r = dma_fence_wait(old, false);
    393.  
    394. 				mutex_lock(&ring->mutex);
    395. 				ring->last_fence = NULL;
    396. 				dma_fence_put(old);
    397. 				mutex_unlock(&ring->mutex);
    398.  
    399. 				if (kthread_should_stop() || r) {
    400. 					// will call fence_ops.release directly to free the fence.
    401. 					dma_fence_put(&fence->job);
    402. 					continue;
    403. 				}
    404.  
    405. 				// if overlap happened, there must be a congruences on seq and old->seqno,which means seq≡ old->seqno mod(num_hw_submission * 2)
    406. 				// this implies seq = q*(num_hw_submission * 2) + old->seqno. q=1 typically.
    407. 				diff = seq - old->seqno;
    408. 				printk("%s line %d, fence wokenup, wakeseqno %lld, new adding seq %d, slot %d, diff %d, waken interval %lld, latestseq %d, avail %d.\n",
    409. 				       __func__, __LINE__, old->seqno, seq, seq & ring->num_fences_mask, diff, old->seqno - oldwaitseqno,
    410. 				       ring->sync_seq, fencedrv_get_ring_avail());
    411.  
    412. 				if (diff != num_hw_submission * 2) {
    413. 					pr_err("%s line %d, fatal error, diff not match totoal ring.\n",
    414. 					       __func__, __LINE__);
    415. 				}
    416.  
    417. 				oldwaitseqno = old->seqno;
    418. 			}
    419. 		}
    420.  
    421. #if 0
    422. 		printk("%s line %d, fence emit, seqno %lld, seq %d, slot %d.\n",
    423. 		       __func__, __LINE__, fence->seqno, seq, seq & ring->num_fences_mask);
    424. #endif
    425. 		rcu_assign_pointer(*ptr, dma_fence_get(&fence->job));
    426.  
    427. 		// because no outer usage of fence, so put it here for free ok.
    428. 		dma_fence_put(&fence->job);
    429. #endif
    430. 	}
    431.  
    432. 	set_current_state(TASK_RUNNING);
    433. 	return 0;
    434. }
    435.  
    436. void work_timer_fn(struct timer_list *timer)
    437. {
    438. 	uint32_t seqno_next = 0;
    439. 	uint32_t seq, last_seq;
    440. 	int r;
    441.  
    442. 	do {
    443. 		last_seq = atomic_read(&ring->last_seq);
    444. 		seq = *ring->cpu_addr;
    445. 	} while (atomic_cmpxchg(&ring->last_seq, last_seq, seq) != last_seq);
    446.  
    447. 	if (unlikely(seq == last_seq)) {
    448. 		goto end;
    449. 	}
    450.  
    451. 	assert(seq > last_seq);
    452.  
    453. 	last_seq &= ring->num_fences_mask;
    454. 	seq &= ring->num_fences_mask;
    455.  
    456. 	do {
    457. 		struct dma_fence *fence, **ptr;
    458.  
    459. 		++last_seq;
    460. 		last_seq &= ring->num_fences_mask;
    461. 		ptr = &ring->fences[last_seq];
    462. 		fence = rcu_dereference_protected(*ptr, 1);
    463.  
    464. 		RCU_INIT_POINTER(*ptr, NULL);
    465. 		if (!fence) {
    466. 			continue;
    467. 		}
    468.  
    469. 		if (seqno_next == 0 || seqno_next == fence->seqno) {
    470. 			seqno_next = fence->seqno + 1;
    471. 		} else { /*if (seqno_next != 0 && seqno_next != fence->seqno)*/
    472. 			pr_err("%s line %d, seqno is not continue, exptect %d, actual %lld.\n",
    473. 			       __func__, __LINE__, seqno_next, fence->seqno);
    474. 		}
    475.  
    476. 		r = dma_fence_signal(fence);
    477. 		if (r) {
    478. 			pr_err("%s line %d, fence already signaled.\n",
    479. 			       __func__, __LINE__);
    480. 			continue;
    481. 			//BUG();
    482. 		}
    483.  
    484. 		dma_fence_put(fence);
    485. 	} while (last_seq != seq);
    486. end:
    487. 	pr_err("%s line %d, work timer triggerd.\n", __func__, __LINE__);
    488. 	mod_timer(timer, jiffies + HZ / 10);
    489. }
    490.  
    491. void gpu_process_thread(struct timer_list *timer)
    492. {
    493. 	uint32_t seq, oldseq;
    494.  
    495. 	seq = ring->sync_seq;
    496. 	oldseq = fence_seq;
    497.  
    498. 	// trigger a job done on device.
    499. 	if (fence_seq == 0) {
    500. 		if (seq > 6)
    501. 			fence_seq = seq - 4;
    502. 	} else if ((seq - fence_seq) > 10) {
    503. 		fence_seq += (seq - fence_seq) / 2;
    504. 		assert(fence_seq > oldseq);
    505. 	}
    506.  
    507. 	printk("%s line %d, timer trigger job, latest consume fence %d.\n",
    508. 	       __func__, __LINE__, fence_seq);
    509.  
    510. 	mod_timer(timer, jiffies + HZ / 2);
    511. }
    512.  
    513. static int fencedrv_wait_empty(void)
    514. {
    515. 	uint64_t seq = READ_ONCE(ring->sync_seq);
    516. 	struct dma_fence *fence, **ptr;
    517. 	int r;
    518.  
    519. 	if (!seq)
    520. 		return 0;
    521.  
    522. 	fence_seq = seq;
    523. 	ptr = &ring->fences[seq & ring->num_fences_mask];
    524. 	rcu_read_lock();
    525.  
    526. 	fence = rcu_dereference(*ptr);
    527. 	if (!fence || !dma_fence_get_rcu(fence)) {
    528. 		rcu_read_unlock();
    529. 		return 0;
    530. 	}
    531. 	rcu_read_unlock();
    532.  
    533. 	r = dma_fence_wait(fence, false);
    534.  
    535. 	printk("%s line %d, wait last fence %lld, seq %lld, r %d.\n", \
    536. 	       __func__, __LINE__, fence->seqno, seq, r);
    537. 	dma_fence_put(fence);
    538.  
    539. 	return r;
    540. }
    541.  
    542. static int __init fencedrv_init(void)
    543. {
    544. 	if ((num_hw_submission & (num_hw_submission - 1)) != 0) {
    545. 		pr_err("%s line %d, num_hw_submission must be power of two.\n",
    546. 		       __func__, __LINE__);
    547. 		return -1;
    548. 	}
    549.  
    550. 	ring = kzalloc(sizeof(*ring), GFP_KERNEL);
    551. 	if (ring == NULL) {
    552. 		pr_err("%s line %d, alloc fence driver failure.\n",
    553. 		       __func__, __LINE__);
    554. 		return -ENOMEM;
    555. 	}
    556.  
    557. 	// fence_seq is a snap shot of sync_seq for deal with fence batchlly.
    558. 	ring->cpu_addr = &fence_seq;
    559. 	ring->gpu_addr = (uint64_t)&fence_seq;
    560. 	ring->sync_seq = 0;
    561. 	atomic_set(&ring->last_seq, 0);
    562. 	ring->initialized = false;
    563. 	ring->initialized_emit = false;
    564. 	ring->initialized_recv = false;
    565. 	ring->last_fence = NULL;
    566. 	ring->num_fences_mask = num_hw_submission * 2 - 1;
    567. 	init_waitqueue_head(&ring->job_scheduled);
    568.  
    569. 	spin_lock_init(&ring->lock);
    570. 	ring->fences = kcalloc(num_hw_submission * 2, sizeof(void *), GFP_KERNEL);
    571. 	if (!ring->fences) {
    572. 		pr_err("%s line %d, alloc fence buffer failure.\n",
    573. 		       __func__, __LINE__);
    574. 		return -ENOMEM;
    575. 	}
    576.  
    577. 	printk("%s line %d, fence mask 0x%x, num_hw_submission 0x%x.\n",
    578. 	       __func__, __LINE__, ring->num_fences_mask, num_hw_submission);
    579.  
    580. 	job_fence_slab = kmem_cache_create("job_fence_slab", sizeof(struct job_fence), 0,
    581. 	                                   SLAB_HWCACHE_ALIGN, NULL);
    582. 	if (!job_fence_slab) {
    583. 		pr_err("%s line %d, alloc job_fence_slab falure.\n",
    584. 		       __func__, __LINE__);
    585. 		return -ENOMEM;
    586. 	}
    587.  
    588. 	sched_fence_slab = kmem_cache_create("sched_fence_slab", sizeof(struct fence_set), 0,
    589. 	                                     SLAB_HWCACHE_ALIGN, NULL);
    590. 	if (!sched_fence_slab) {
    591. 		pr_err("%s line %d, alloc sched_fence_slab falure.\n",
    592. 		       __func__, __LINE__);
    593. 		return -ENOMEM;
    594. 	}
    595.  
    596. 	mutex_init(&ring->mutex);
    597.  
    598. 	fence_emit_task = kthread_run(fence_emit_task_thread, NULL, "fence_emit");
    599. 	if (IS_ERR(fence_emit_task)) {
    600. 		pr_err("%s line %d, create fence emit tsk failure.\n",
    601. 		       __func__, __LINE__);
    602. 		return -1;
    603. 	}
    604.  
    605. 	fence_recv_task = kthread_run(fence_recv_task_thread, NULL, "fence_recv");
    606. 	if (IS_ERR(fence_recv_task)) {
    607. 		pr_err("%s line %d, create fence recv tsk failure.\n",
    608. 		       __func__, __LINE__);
    609. 		return -1;
    610. 	}
    611.  
    612. 	timer_setup(&ring->timer, gpu_process_thread, TIMER_IRQSAFE);
    613. 	add_timer(&ring->timer);
    614. 	mod_timer(&ring->timer, jiffies + HZ / 2);
    615.  
    616. 	timer_setup(&ring->work_timer, work_timer_fn, TIMER_IRQSAFE);
    617. 	add_timer(&ring->work_timer);
    618. 	mod_timer(&ring->work_timer, jiffies + HZ / 10);
    619.  
    620. 	printk("%s line %d, module init.\n", __func__, __LINE__);
    621.  
    622. 	ring->initialized = true;
    623. 	ring->initialized_emit = true;
    624. 	ring->initialized_recv = true;
    625.  
    626. 	wake_up_process(fence_emit_task);
    627. 	wake_up_process(fence_recv_task);
    628.  
    629. 	return 0;
    630. }
    631.  
    632. static void __exit fencedrv_exit(void)
    633. {
    634. 	printk("%s line %d, module unload task begin.\n", __func__, __LINE__);
    635.  
    636. 	del_timer_sync(&ring->work_timer);
    637.  
    638. 	mutex_lock(&ring->mutex);
    639. 	if ((ring->last_fence != NULL) &&
    640. 	    (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &ring->last_fence->flags))) {
    641. 		ring->initialized_emit = false;
    642. 		dma_fence_signal(ring->last_fence);
    643. 		dma_fence_put(ring->last_fence);
    644. 	}
    645. 	mutex_unlock(&ring->mutex);
    646.  
    647. 	kthread_stop(fence_emit_task);
    648. 	printk("%s line %d, module unload task mid.\n", __func__, __LINE__);
    649.  
    650. 	del_timer_sync(&ring->timer);
    651. 	fencedrv_wait_empty();
    652.  
    653. 	printk("%s line %d, sync wait avail %d.\n", __func__, __LINE__, fencedrv_get_ring_avail());
    654.  
    655. 	wait_event_killable(ring->job_scheduled, fencedrv_get_ring_avail() <= 1);
    656. 	ring->initialized_recv = false;
    657. 	kthread_stop(fence_recv_task);
    658.  
    659. 	printk("%s line %d, module unload task end.\n", __func__, __LINE__);
    660.  
    661. 	ring->initialized = false;
    662. 	rcu_barrier();
    663. 	kmem_cache_destroy(job_fence_slab);
    664. 	kmem_cache_destroy(sched_fence_slab);
    665. 	kfree(ring->fences);
    666. 	kfree(ring);
    667.  
    668. 	fence_emit_task = NULL;
    669. 	fence_recv_task = NULL;
    670.  
    671. 	printk("%s line %d, module unload.\n", __func__, __LINE__);
    672. }
    673.  
    674. module_init(fencedrv_init);
    675. module_exit(fencedrv_exit);
    676. MODULE_LICENSE("GPL");
    677. MODULE_AUTHOR("czl");

    fence的释放顺序

    参考博客

    AMD GPU任务调度（3） —— fence机制_gpu fence-CSDN博客

    Fence_dma fence_笔落梦昙的博客-CSDN博客

    结束

  • 相关阅读:
    力扣hot100——第5天：22括号生成、23合并K个升序链表、31下一个排列
    jQuery实现下拉菜单
    vue内置组件keep-alive的使用
    基于Python语言的Selenium环境搭建
    dm关键字提示报错
    Go简单入门
    RichView Table 表格对齐
    每日一博 - WebHook vs Polling
    【数据库系统概论】实验六 完整性控制
    java（Map接口）
  • 原文地址：https://blog.csdn.net/tugouxp/article/details/133519133