Postgresql源码（92）深入分析HOT更新

0 概述与总结

hot更新已经有几篇分析了，这里是最后一篇（总结性的，前面的可以忽略）。
前面在看update代码时，大部分集中在heap_update上，没有涉及寻找HOT链的逻辑。本篇重点看HOT链是如何使用的。

（总结速查）

（lp=line pointer：页面中等宽数组，每一个指向页面底部的数据区域）

关键步骤总结（no vacuum场景）：

HOT链的头部元素的lp始终保存，索引始终指向这个lp（即使这个lp指向行更新了也只是把数据删了，保留lp指针）
HOT链的中间元素都带HEAP_HOT_UPDATED标记，HOT链的最后一个元素只有HEAP_ONLY_TUPLE标记。
HOT链在更新时有三个关键步骤：
- 1 走索引找到链头lp：table_index_fetch_tuple（下文3.1）
- 2 遍历HOT链确定需要的lp：heap_hot_search_buffer（下文3.2）
- 3 碎片整理，使数据区域更紧凑，会更新lp的指向位置：compactify_tuples（下文3.3）
- 4 使用找到的lp获取页面位置，memcopy数据上去完成update：heap_update（下文3.4）

堆栈：

ExecModifyTable
  ExecProcNode        // ExecProcNode = 0x783005 
  | ExecIndexScan
  |   ExecScan
  |     ExecScanFetch
  |       IndexNext
  |         index_getnext_slot
  |           tid = index_getnext_tid                       // 3.1 总是拿到ip_posid = 130
  |           index_fetch_heap
  |             table_index_fetch_tuple(ItemPointer tid)    // {ip_posid = 130}
  |               heapam_index_fetch_tuple
  |                 heap_hot_search_buffer                  // 3.2 遍历HOT链找旧元组
  |                 heap_page_prune_opt                     // 3.3 碎片整理
  |                   heap_page_prune
  |                     heap_page_prune_execute
  |                       PageRepairFragmentation
  |                         compactify_tuples
  ExecUpdate
    ExecUpdateAct
      table_tuple_update
        heapam_tuple_update
          heap_update                                       // 3.4 更新
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

1 分析用例

照常先给出分析用例。

-- 测试表，单页可以放136条数据
drop table testbl;
create table testbl(i int primary key not null, id int not null, info varchar(200) not null);
alter table testbl set (autovacuum_enabled = false);
insert into testbl select generate_series(1,130), (random()*100)::integer, repeat('DUfw',(5)::integer);


select * from testbl limit 10;
 i  | id |         info         
----+----+----------------------
  1 | 57 | DUfwDUfwDUfwDUfwDUfw
  2 |  2 | DUfwDUfwDUfwDUfwDUfw
  3 | 29 | DUfwDUfwDUfwDUfwDUfw
  4 | 37 | DUfwDUfwDUfwDUfwDUfw
  5 |  2 | DUfwDUfwDUfwDUfwDUfw
  6 | 44 | DUfwDUfwDUfwDUfwDUfw
  7 | 53 | DUfwDUfwDUfwDUfwDUfw
  8 | 24 | DUfwDUfwDUfwDUfwDUfw
  9 | 49 | DUfwDUfwDUfwDUfwDUfw
 10 | 17 | DUfwDUfwDUfwDUfwDUfw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

2 HOT更新实验

2.1 总结：等宽更新

HOT多次更新后，发现HOT链会复用元组，并不会一直延长。
HOT根节点（本例中的lp=130）不变，复用后续的节点。
复用节点带HEAP_ONLY_TUPLE | HEAP_HOT_UPDATED标记。

UPDATE	lp=130	lp=131	lp=132
第一次	原tuple：HEAP_HOT_UPDATED	新tuple：HEAP_ONLY_TUPLE
第二次		原tuple：HEAP_ONLY_TUPLE HEAP_HOT_UPDATED	新tuple：HEAP_ONLY_TUPLE
第三次		新tuple：HEAP_ONLY_TUPLE	原tuple：HEAP_ONLY_TUPLE HEAP_HOT_UPDATED
第四次		原tuple：HEAP_ONLY_TUPLE HEAP_HOT_UPDATED	新tuple：HEAP_ONLY_TUPLE
第五次		新tuple：HEAP_ONLY_TUPLE	原tuple：HEAP_ONLY_TUPLE HEAP_HOT_UPDATED

数组区域状态（等宽更新，对应上表）

注意uppdate都是使用lp有效的位置，用之前先做碎片整理，把有效的向下移动，填充到删除的地方。然后再insert。
在这里插入图片描述

2.1 总结：不等宽更新

数组区域状态（不等宽更新）

注意第四次更新和第五次更新，新数据更宽了，可以明显看到碎片整理的过程：

第五次更新时，先把132的数据向下移动到888-967；然后再对132的数据进行更新；更新后132被删除；131被复用，放在了页面的upper指针+数据大小的位置。

实验：更新前xid=8169

SELECT lp, lp_off, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask FROM heap_page_items(get_raw_page('testbl', 0));

 lp  | lp_off | t_xmin | t_xmax | t_field3 | t_ctid  | t_infomask2 | t_infomask 
-----+--------+--------+--------+----------+---------+-------------+------------
 130 |    912 |   8169 |      0 |        0 | (0,130) |           3 |       2050

============
t_infomask
============
2050(0x802) = HEAP_XMAX_INVALID | HEAP_HASVARWIDTH

============
t_infomask2(11 bits for number of attributes)
============
number_of_ = 3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

实验：第一次更新xid=8170

update testbl set info = 'DDDDDDDDDDDDDDDDDDD1' where i = 130;

SELECT lp, lp_off, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask FROM heap_page_items(get_raw_page('testbl', 0));
 lp  | lp_off | t_xmin | t_xmax | t_field3 | t_ctid  | t_infomask2 | t_infomask 
-----+--------+--------+--------+----------+---------+-------------+------------
 130 |    912 |   8169 |   8170 |        0 | (0,131) |       16387 |        258
 131 |    856 |   8170 |      0 |        0 | (0,131) |       32771 |      10242
 
============
t_infomask
============
258(0x102) = HEAP_XMIN_COMMITTED | HEAP_HASVARWIDTH
10242(0x2802) = HEAP_UPDATED | HEAP_XMAX_INVALID | HEAP_HASVARWIDTH

============
t_infomask2
============
16387(0x4003) = HEAP_HOT_UPDATED | 3attributes
32771(0x8003) = HEAP_ONLY_TUPLE | 3attributes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

实验：第二次更新xid=8171

update testbl set info = 'DDDDDDDDDDDDDDDDDDD2' where i = 130;

SELECT lp, lp_off, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask FROM heap_page_items(get_raw_page('testbl', 0));
 lp  | lp_off | t_xmin | t_xmax | t_field3 | t_ctid  | t_infomask2 | t_infomask 
-----+--------+--------+--------+----------+---------+-------------+------------
 130 |    131 |        |        |          |         |             |           
 131 |    912 |   8170 |   8171 |        0 | (0,132) |       49155 |       8450
 132 |    856 |   8171 |      0 |        0 | (0,132) |       32771 |      10242

============
t_infomask
============
8450(0x2102) = HEAP_UPDATED | HEAP_XMIN_COMMITTED | HEAP_HASVARWIDTH
10242(0x2802) = HEAP_UPDATED | HEAP_XMAX_INVALID | HEAP_HASVARWIDTH

============
t_infomask2
============
49155(0xC003) = HEAP_ONLY_TUPLE | HEAP_HOT_UPDATED | 3attributes
32771(0x8003) = HEAP_ONLY_TUPLE | 3attributes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

实验：第三次更新xid=8172

update testbl set info = 'DDDDDDDDDDDDDDDDDDD3' where i = 130;

SELECT lp, lp_off, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask FROM heap_page_items(get_raw_page('testbl', 0));
 lp  | lp_off | t_xmin | t_xmax | t_field3 | t_ctid  | t_infomask2 | t_infomask 
-----+--------+--------+--------+----------+---------+-------------+------------
 130 |    132 |        |        |          |         |             |           
 131 |    856 |   8172 |      0 |        0 | (0,131) |       32771 |      10242
 132 |    912 |   8171 |   8172 |        0 | (0,131) |       49155 |       8450
 
============
t_infomask
============
10242(0x2802) = HEAP_UPDATED | HEAP_XMAX_INVALID | HEAP_HASVARWIDTH
8450(0x2102) = HEAP_UPDATED | HEAP_XMIN_COMMITTED | HEAP_HASVARWIDTH

============
t_infomask2
============
49155(0xC003) = HEAP_ONLY_TUPLE | HEAP_HOT_UPDATED | 3attributes
32771(0x8003) = HEAP_ONLY_TUPLE | 3attributes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

实验：第四次更新xid=8173

update testbl set info = 'DDDDDDDDDDDDDDDDDDD4' where i = 130;

SELECT lp, lp_off, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask FROM heap_page_items(get_raw_page('testbl', 0));
 lp  | lp_off | t_xmin | t_xmax | t_field3 | t_ctid  | t_infomask2 | t_infomask 
-----+--------+--------+--------+----------+---------+-------------+------------
 130 |    131 |        |        |          |         |             |           
 131 |    912 |   8172 |   8173 |        0 | (0,132) |       49155 |       8450
 132 |    856 |   8173 |      0 |        0 | (0,132) |       32771 |      10242

============
t_infomask
============
8450(0x2102) = HEAP_UPDATED | HEAP_XMIN_COMMITTED | HEAP_HASVARWIDTH
10242(0x2802) = HEAP_UPDATED | HEAP_XMAX_INVALID | HEAP_HASVARWIDTH

============
t_infomask2
============
49155(0xC003) = HEAP_ONLY_TUPLE | HEAP_HOT_UPDATED | 3attributes
32771(0x8003) = HEAP_ONLY_TUPLE | 3attributes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

实验：第五次更新xid=8174

update testbl set info = 'DDDDDDDDDDDDDDDDDDD5' where i = 130;

SELECT lp, lp_off, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask FROM heap_page_items(get_raw_page('testbl', 0));
 lp  | lp_off | t_xmin | t_xmax | t_field3 | t_ctid  | t_infomask2 | t_infomask 
-----+--------+--------+--------+----------+---------+-------------+------------
 130 |    132 |        |        |          |         |             |           
 131 |    856 |   8174 |      0 |        0 | (0,131) |       32771 |      10242
 132 |    912 |   8173 |   8174 |        0 | (0,131) |       49155 |       8450

============
t_infomask
============
10242(0x2802) = HEAP_UPDATED | HEAP_XMAX_INVALID | HEAP_HASVARWIDTH
8450(0x2102) = HEAP_UPDATED | HEAP_XMIN_COMMITTED | HEAP_HASVARWIDTH

============
t_infomask2
============
49155(0xC003) = HEAP_ONLY_TUPLE | HEAP_HOT_UPDATED | 3attributes
32771(0x8003) = HEAP_ONLY_TUPLE | 3attributes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

3 场景分析：lp=130空、lp=131删除、lp=132有效

当前HOT链：130（重定向）—>132（有效）
预期发生：碎片整理：132准备转移到131的位置；新增数据到132原来的位置、用131指向新增数据。

3.1 索引扫描

从顶层ExecModifyTable进入索引扫描部分，因为关掉了VAUUM，索引总是返回130：index_getnext_tid

ExecModifyTable
  ExecProcNode        // ExecProcNode = 0x783005 
    ExecIndexScan
      ExecScan
        ExecScanFetch
          IndexNext
            index_getnext_slot
              tid = index_getnext_tid                       // 3.1 总是拿到ip_posid = 130  <<<<
              index_fetch_heap
                table_index_fetch_tuple(ItemPointer tid)    // {ip_posid = 130}
                  heapam_index_fetch_tuple
                    heap_hot_search_buffer                  // 3.2 遍历HOT链找旧元组
                    heap_page_prune_opt                     // 3.3 碎片整理
1
2
3
4
5
6
7
8
9
10
11
12
13

3.2 遍历HOT链找旧元组

再用130去找元组：heap_hot_search_buffer

bool
heap_hot_search_buffer(
			ItemPointer tid,  // {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 130}
			Relation relation, Buffer buffer,
			Snapshot snapshot, HeapTuple heapTuple,
			bool *all_dead, bool first_call)
{
	...
	...	
	/* Scan through possible multiple members of HOT-chain */
	for (;;)
	{
		ItemId		lp;

		/* check for bogus TID */
		if (offnum < FirstOffsetNumber || offnum > PageGetMaxOffsetNumber(dp))
			break;
		
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

【第一轮循环读到HOT链130，找到132】

这里比较重要，offnum现在还是130，但是lp拿出来就直接是132了：
ItemIdData = {lp_off = 132, lp_flags = 2, lp_len = 0}
这里的lp_flags=2表示LP_REDIRECT，重定向到132。
#define LP_UNUSED 0 /* unused (should always have lp_len=0) */
#define LP_NORMAL 1 /* used (should always have lp_len>0) */
#define LP_REDIRECT 2 /* HOT redirect (should have lp_len=0) */
#define LP_DEAD 3 /* dead, may or may not have storage */

在这里插入图片描述

【第二轮循环读到HOT链132，找到数据位置】

第二次进入循环后，offnum=132拿到的lp：
ItemIdData = {lp_off = 912, lp_flags = 1, lp_len = 53}（LP_NORMAL）

现在的912就是指向数据区域了。

		lp = PageGetItemId(dp, offnum);

		/* check for unused, dead, or redirected items */
		if (!ItemIdIsNormal(lp))
		{
			/* We should only see a redirect at start of chain */
			if (ItemIdIsRedirected(lp) && at_chain_start)
			{
				/* Follow the redirect */
				offnum = ItemIdGetRedirect(lp);
1
2
3
4
5
6
7
8
9
10

使用lp=132拿到offnum=132，继续下一轮循环继续遍历HOT链。

				at_chain_start = false;
				continue;
			}
			/* else must be end of chain */
			break;
		}

1
2
3
4
5
6
7

遍历完HOT链出循环，开始读132的数据部分，拼接元组。

		heapTuple->t_data = (HeapTupleHeader) PageGetItem(dp, lp);
		heapTuple->t_len = ItemIdGetLength(lp);
		heapTuple->t_tableOid = RelationGetRelid(relation);
		ItemPointerSet(&heapTuple->t_self, blkno, offnum);

		if (!skip)
		{
			/* If it's visible per the snapshot, we must return it */
			valid = HeapTupleSatisfiesVisibility(heapTuple, snapshot, buffer);
			HeapCheckForSerializableConflictOut(valid, relation, heapTuple,
												buffer, snapshot);

			if (valid)
			{
				ItemPointerSetOffsetNumber(tid, offnum);
				PredicateLockTID(relation, &heapTuple->t_self, snapshot,
								 HeapTupleHeaderGetXmin(heapTuple->t_data));
				if (all_dead)
					*all_dead = false;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

找到132返回。

				return true;
			}
		}
		skip = false;
1
2
3
4

3.3 碎片整理

将132移动到131的位置上，因为131删掉已经是空洞了。

这里就不展开分析了。记录下函数堆栈。

ExecModifyTable
  ExecProcNode        // ExecProcNode = 0x783005 
    ExecIndexScan
      ExecScan
        ExecScanFetch
          IndexNext
            index_getnext_slot
              tid = index_getnext_tid                       // 总是拿到ip_posid = 130  <<<<
              index_fetch_heap
                table_index_fetch_tuple(ItemPointer tid)    // {ip_posid = 130}
                  heapam_index_fetch_tuple
                    heap_hot_search_buffer                  // 3.2 遍历HOT链找旧元组
                    heap_page_prune_opt                     // 3.3 碎片整理
                      heap_page_prune
                        heap_page_prune_execute
                          PageRepairFragmentation
                            compactify_tuples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

3.4 开始更新heap_update

ExecModifyTable
  ExecProcNode        // ExecProcNode = 0x783005 
  | ExecIndexScan
  |   ExecScan
  |     ExecScanFetch
  |       IndexNext
  |         index_getnext_slot
  |           tid = index_getnext_tid                       // 总是拿到ip_posid = 130  <<<<
  |           index_fetch_heap
  |             table_index_fetch_tuple(ItemPointer tid)    // {ip_posid = 130}
  |               heapam_index_fetch_tuple
  |                 heap_hot_search_buffer                  // 3.2 遍历HOT链找旧元组
  |                 heap_page_prune_opt                     // 3.3 碎片整理
  |                   heap_page_prune
  |                     heap_page_prune_execute
  |                       PageRepairFragmentation
  |                         compactify_tuples
  ExecUpdate
    ExecUpdateAct
      table_tuple_update
        heapam_tuple_update
          heap_update
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

1 拿到数据地址

ItemId = {lp_off = 912, lp_flags = 1, lp_len = 53}

lp = PageGetItemId(page, ItemPointerGetOffsetNumber(otid));
1

2 拼接旧元组：

HeapTupleData = {t_len = 53, t_self = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 132}, t_tableOid = 32946, t_data = 0x2aaaab4ae610}

3 新元组位置未确定：

HeapTupleData = {t_len = 53, t_self = {ip_blkid = {bi_hi = 65535, bi_lo = 65535}, ip_posid = 0}, t_tableOid = 32946, t_data = 0x26fe440}

	oldtup.t_tableOid = RelationGetRelid(relation);
	oldtup.t_data = (HeapTupleHeader) PageGetItem(page, lp);
	oldtup.t_len = ItemIdGetLength(lp);
	oldtup.t_self = *otid;
	
	newtup->t_tableOid = RelationGetRelid(relation);
1
2
3
4
5
6

4 判断旧元组可见性

result = TM_Ok

result = HeapTupleSatisfiesUpdate(&oldtup, cid, buffer);
1

5 同页更新使用use_hot_update

旧的加：HEAP_HOT_UPDATED
新的加：HEAP_ONLY_TUPLE

1: *newtup = {t_len = 53, t_self = {ip_blkid = {bi_hi = 65535, bi_lo = 65535}, ip_posid = 0}, t_tableOid = 32946, t_data = 0x26fe440}
3: oldtup = {t_len = 53, t_self = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 132}, t_tableOid = 32946, t_data = 0x2aaaab4ae610}
1
2

6 insert插入新tuple

RelationPutHeapTuple执行insert操作，需要先找到插入位置。

RelationPutHeapTuple
	PageAddItem
1
2

核心逻辑（重要）

正向遍历itemid，所以，数据区域是反向遍历的。
找到空闲的lp131，然后132经过碎片整理下移到131的位置上了，原来132的位置在最上面空出来了，申请这个位置使用。

			for (offsetNumber = FirstOffsetNumber;
				 offsetNumber < limit;	/* limit is maxoff+1 */
				 offsetNumber++)
			{
				itemId = PageGetItemId(phdr, offsetNumber);

				/*
				 * We check for no storage as well, just to be paranoid;
				 * unused items should never have storage.  Assert() that the
				 * invariant is respected too.
				 */
				Assert(ItemIdIsUsed(itemId) || !ItemIdHasStorage(itemId));

				if (!ItemIdIsUsed(itemId) && !ItemIdHasStorage(itemId))
					break;
			}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

在这里插入图片描述

相关阅读:
CP AUTOSAR值Ethernet Test Module(TC8)
Hystirx限流：信号量隔离和线程池隔离
 VoLTE端到端业务详解 | 空口主要信令过程
 海格里斯HEGERLS托盘式四向穿梭车批发定制｜24小时全自动无人化立体仓库批量托盘作业
 当开源项目 Issue 遇到了 DevChat
Dagger2系列详解(三)：@privider 依赖
 Redis持久化AOF/RDB
C/C++ 数据结构 - 栈
 TFT-eSPI入门使用教程
 进阶JAVA篇- BigDecimal 类的常用API（四）
原文地址：https://blog.csdn.net/jackgo73/article/details/128123393