• 深入linux内核架构--slab分配器（建议收藏）

    简介：malloc对于大家来说应该都不陌生了，这是系统库给我们提供了申请指定大小内存的函数，之前介绍的伙伴系统，只能以页的方式申请内存，对于小块(小于一页)内存的申请我们就得通过自定义的库函数来实现相关需求，所以在用户空间层面诞生了诸如ptmalloc(glibc)，tcmalloc(google)，jemalloc(facebook)等优秀的内存分配库。但是这些库内核没法使用，且内核也有大量申请小块内存的需求，诸如管理dentry，inode，fs_struct，page，task_struct等等一系列内核对象。所以内核提出了slab分配器，用来管理内核中小块内存分配，而cpu cache也是配合slab使用的，有时候也把slab称为缓存。

    内核中内存管理

    对于内核来说，slab主要包括kmalloc及kfree两个函数来分配及释放小块内存：
    kmalloc(size, flags)：分配长度为size字节的一个内存区，并返回指向该内存区起始void指针，如果没有足够内存，返回NULL。

    kfree(ptr)：释放ptr指向的内存区。

    对于内核开发者还可以通过kmem_cache_create创建一个缓存kmem_cache对象；
    通过kmem_cache_alloc、kmem_cache_alloc_node提供特定类型的内核缓存对象申请。他们最终都会调用到slab_alloc。所以主要的slab操作都在slab_alloc函数中。

    slab缓存由两部分组成：保存管理性数据的缓存对象和保存被管理对象的各个slab对象

    【文章福利】小编推荐自己的Linux内核技术交流群： 【977878001】整理一些个人觉得比较好得学习书籍、视频资料共享在群文件里面，有需要的可以自行添加哦！！！前100进群领取，额外赠送一份 价值699的内核资料包（含视频教程、电子书、实战项目及代码）

    内核资料直通车：Linux内核源码技术学习路线+视频教程代码资料

    学习直通车：Linux内核源码/内存调优/文件系统/进程管理/设备驱动/网络协议栈-学习视频教程-腾讯课堂

    slab cache
    上图中缓存即为kmem_cache，slab即为page页帧，缓存对象即为void指针。一个kmem_cache会在不同的内存节点管理很多页帧，这些页帧在各个内存节点被划分为3类：部分空闲，全部空闲以及全部分配。

    每个缓存kmem_cache对象只负责一种slab对象类型的管理，各个kmem_cache缓存中slab对象大小各不相同，由创建的时候指定，而缓存会根据指定的slab对象大小根据cpu cacheline 或者void*指针大小进行对齐，然后根据一个公式计算一个合适的gfporder来确定每次申请的内存页帧的数量。其计算方法为：

    1. 依次递增gfporder，这样关联页数就为2^gfporder，对应的字节数为PAGE_SIZE << gfporder；
    2. PAGE_SIZE << gfporder = head + num * size + left_over
    3. 当left_over * 8 <=  PAGE_SIZE << gfporder 时就决定是这个gfporder，因为这是一个可以接受的碎片大小。

    对应的计算函数为calculate_slab_order，相关代码如下：

    1. /**
    2.  * calculate_slab_order - calculate size (page order) of slabs
    3.  * @cachep: pointer to the cache that is being created
    4.  * @size: size of objects to be created in this cache.
    5.  * @flags: slab allocation flags
    6.  *
    7.  * Also calculates the number of objects per slab.
    8.  *
    9.  * This could be made much more intelligent.  For now, try to avoid using
    10.  * high order pages for slabs.  When the gfp() functions are more friendly
    11.  * towards high-order requests, this should be changed.
    12.  */
    13. static size_t calculate_slab_order(struct kmem_cache *cachep,
    14.                 size_t size, slab_flags_t flags)
    15. {
    16.     size_t left_over = 0;
    17.     int gfporder;
    18.     for (gfporder = 0; gfporder <= KMALLOC_MAX_ORDER; gfporder++) {
    19.         unsigned int num;
    20.         size_t remainder;
    21.         num = cache_estimate(gfporder, size, flags, &remainder);
    22.         if (!num)
    23.             continue;
    24.         /* Can't handle number of objects more than SLAB_OBJ_MAX_NUM */
    25.         if (num > SLAB_OBJ_MAX_NUM)
    26.             break;
    27.         if (flags & CFLGS_OFF_SLAB) {
    28.             struct kmem_cache *freelist_cache;
    29.             size_t freelist_size;
    30.             freelist_size = num * sizeof(freelist_idx_t);
    31.             freelist_cache = kmalloc_slab(freelist_size, 0u);
    32.             if (!freelist_cache)
    33.                 continue;
    34.             /*
    35.              * Needed to avoid possible looping condition
    36.              * in cache_grow_begin()
    37.              */
    38.             if (OFF_SLAB(freelist_cache))
    39.                 continue;
    40.             /* check if off slab has enough benefit */
    41.             if (freelist_cache->size > cachep->size / 2)
    42.                 continue;
    43.         }
    44.         /* Found something acceptable - save it away */
    45.         cachep->num = num;
    46.         cachep->gfporder = gfporder;
    47.         left_over = remainder;
    48.         /*
    49.          * A VFS-reclaimable slab tends to have most allocations
    50.          * as GFP_NOFS and we really don't want to have to be allocating
    51.          * higher-order pages when we are unable to shrink dcache.
    52.          */
    53.         if (flags & SLAB_RECLAIM_ACCOUNT)
    54.             break;
    55.         /*
    56.          * Large number of objects is good, but very large slabs are
    57.          * currently bad for the gfp()s.
    58.          */
    59.         if (gfporder >= slab_max_order)
    60.             break;
    61.         /*
    62.          * Acceptable internal fragmentation?
    63.          */
    64.         if (left_over * 8 <= (PAGE_SIZE << gfporder))
    65.             break;
    66.     }
    67.     return left_over;
    68. }
    69. /*
    70.  * Calculate the number of objects and left-over bytes for a given buffer size.
    71.  */
    72. static unsigned int cache_estimate(unsigned long gfporder, size_t buffer_size,
    73.         slab_flags_t flags, size_t *left_over)
    74. {
    75.     unsigned int num;
    76.     size_t slab_size = PAGE_SIZE << gfporder;
    77.     /*
    78.      * The slab management structure can be either off the slab or
    79.      * on it. For the latter case, the memory allocated for a
    80.      * slab is used for:
    81.      *
    82.      * - @buffer_size bytes for each object
    83.      * - One freelist_idx_t for each object
    84.      *
    85.      * We don't need to consider alignment of freelist because
    86.      * freelist will be at the end of slab page. The objects will be
    87.      * at the correct alignment.
    88.      *
    89.      * If the slab management structure is off the slab, then the
    90.      * alignment will already be calculated into the size. Because
    91.      * the slabs are all pages aligned, the objects will be at the
    92.      * correct alignment when allocated.
    93.      */
    94.     if (flags & (CFLGS_OBJFREELIST_SLAB | CFLGS_OFF_SLAB)) {
    95.         num = slab_size / buffer_size;
    96.         *left_over = slab_size % buffer_size;
    97.     } else {
    98.         num = slab_size / (buffer_size + sizeof(freelist_idx_t));
    99.         *left_over = slab_size %
    100.             (buffer_size + sizeof(freelist_idx_t));
    101.     }
    102.     return num;
    103. }

    系统中所有的缓存都保存在一个双链表中，这使得内核可以遍历所有的缓存，这主要用于缩减分配给内存的数量，常见的场景就是：dentry及inode slab缓存的回收，当机器物理内存不足时就会缩减这一部分内存占用(这一部分内存被称为SReclaimable，可以通过cat /proc/meminfo查看)。

    基本结构

    kmem_cache数据结构代表一个slab 缓存，其中有一些缓存元信息包括：缓存名，缓存对象大小，关联的内存页帧数，着色信息等等；还有一个__per_cpu array_cache用于表示该缓存在各个CPU中的slab对象；kmem_cache_node用于管理各个内存节点上slab对象的分配。

    array_cache是一个per_cpu数组，所以访问不需要加锁，是与cpu cache打交道的直接数据结构，每次获取空闲slab对象时都是通过entry[avail--]去获取，当avail==0时，又从kmem_cache_node中获取batchcount个空闲对象到array_cache中。

    kmem_cache_node用于管理slab(实际对象存储伙伴页帧)，其会管理三个slab列表，部分空闲partial，全部空闲empty，全部占用full。array_cache获取batchcount空闲对象时，先尝试从partial分配，如果不够则再从empty分配剩余对象，如果都不够，则需要grow分配新的slab页帧

    page页帧，这个就不必多说了，这是物理存储地址，是一个union结构，当被用作slab时，会初始化一下slab管理数据，诸如起始object地址s_mem，lru缓存节点，是否被激活active，关联到的kmem_cache以及freelist空闲对象数组(是一个void*指针，其实存的是char or short数组)。

    具体数据结构如下：

    1. struct kmem_cache {
    2. /* 0) per-CPU数据，在每次分配/释放期间都会访问 */
    3.     struct array_cache __percpu *cpu_cache;  // 每个cpu中的slab对象
    4.  
    5. /* 1) Cache tunables. Protected by slab_mutex */
    6.     unsigned int batchcount; // 当__percpu  cpu_cache为空时，从缓存slab中获取的对象数目，它还表示缓存增长时分配的对象数目。初始时为1，后续会调整。
    7.     unsigned int limit; // __percpu  cpu_cache中的对象数目上限，当slab free达到limit时，需要将array_caches中的部分obj返回到kmem_cache_node的页帧中。
    8.     unsigned int shared;
    9.  
    10.     unsigned int size; // slab中的每个对象大小
    11.     struct reciprocal_value reciprocal_buffer_size;
    12. /* 2) touched by every alloc & free from the backend */
    13.  
    14.     slab_flags_t flags;     /* constant flags */
    15.     unsigned int num;       /* # of objs per slab */
    16.  
    17. /* 3) cache_grow/shrink */
    18.     /* order of pgs per slab (2^n) */
    19.     unsigned int gfporder; // slab关联页数
    20.  
    21.     /* force GFP flags, e.g. GFP_DMA */
    22.     gfp_t allocflags;  
    23.  
    24.     size_t colour;          /* cache colouring range */
    25.     unsigned int colour_off;    /* colour offset */
    26.     struct kmem_cache *freelist_cache; // 空闲对象管理
    27.     unsigned int freelist_size; // 空闲对象数量
    28.  
    29.     /* constructor func */
    30.     void (*ctor)(void *obj); // 这个在2.6之后已经废弃了
    31.  
    32. /* 4) cache creation/removal */
    33.     const char *name;
    34.     struct list_head list;
    35.     int refcount;
    36.     int object_size;
    37.     int align;
    38.  
    39. /* 5) statistics */
    40. #ifdef CONFIG_DEBUG_SLAB
    41.     unsigned long num_active;
    42.     unsigned long num_allocations;
    43.     unsigned long high_mark;
    44.     unsigned long grown;
    45.     unsigned long reaped;
    46.     unsigned long errors;
    47.     unsigned long max_freeable;
    48.     unsigned long node_allocs;
    49.     unsigned long node_frees;
    50.     unsigned long node_overflow;
    51.     atomic_t allochit;
    52.     atomic_t allocmiss;
    53.     atomic_t freehit;
    54.     atomic_t freemiss;
    55. #ifdef CONFIG_DEBUG_SLAB_LEAK
    56.     atomic_t store_user_clean;
    57. #endif
    58.  
    59.     /*
    60.      * If debugging is enabled, then the allocator can add additional
    61.      * fields and/or padding to every object. 'size' contains the total
    62.      * object size including these internal fields, while 'obj_offset'
    63.      * and 'object_size' contain the offset to the user object and its
    64.      * size.
    65.      */
    66.     int obj_offset;
    67. #endif /* CONFIG_DEBUG_SLAB */
    68.  
    69. #ifdef CONFIG_MEMCG
    70.     struct memcg_cache_params memcg_params;
    71. #endif
    72. #ifdef CONFIG_KASAN
    73.     struct kasan_cache kasan_info;
    74. #endif
    75.  
    76. #ifdef CONFIG_SLAB_FREELIST_RANDOM
    77.     unsigned int *random_seq;
    78. #endif
    79.  
    80.     unsigned int useroffset;    /* Usercopy region offset */
    81.     unsigned int usersize;      /* Usercopy region size */
    82.  
    83.     struct kmem_cache_node *node[MAX_NUMNODES];  // 每个内存节点上的slab对象信息，每个node上包括部分空闲，全满以及全部空闲三个队列
    84. };
    85.  
    86. struct array_cache {
    87.     unsigned int avail; // 保存了当前array中的可用数目
    88.     unsigned int limit; // 同上
    89.     unsigned int batchcount; // 同上
    90.     unsigned int touched; // 缓存收缩时置0，缓存移除对象时置1，使得内核能确认在上一次收缩之后是否被访问过
    91.     void *entry[];  /*
    92.              * Must have this definition in here for the proper
    93.              * alignment of array_cache. Also simplifies accessing
    94.              * the entries.
    95.              */
    96. };
    97.  
    98. struct kmem_cache_node {
    99.     spinlock_t list_lock;
    100.  
    101. #ifdef CONFIG_SLAB
    102.     struct list_head slabs_partial; /* partial list first, better asm code */
    103.     struct list_head slabs_full;
    104.     struct list_head slabs_free;
    105.     unsigned long total_slabs;  /* length of all slab lists */
    106.     unsigned long free_slabs;   /* length of free slab list only */
    107.     unsigned long free_objects;
    108.     unsigned int free_limit;
    109.     unsigned int colour_next;   /* Per-node cache coloring */
    110.     struct array_cache *shared; /* shared per node */
    111.     struct alien_cache **alien; /* on other nodes */
    112.     unsigned long next_reap;    /* updated without locking */
    113.     int free_touched;       /* updated without locking */
    114. #endif
    115.  
    116. #ifdef CONFIG_SLUB
    117.     unsigned long nr_partial;
    118.     struct list_head partial;
    119. #ifdef CONFIG_SLUB_DEBUG
    120.     atomic_long_t nr_slabs;
    121.     atomic_long_t total_objects;
    122.     struct list_head full;
    123. #endif
    124. #endif
    125. };

    kmalloc

    kmalloc的基本调用结构如下：

    主要包括两个操作：

    1. 从kmalloc_caches中获取kmem_cache(kmalloc_cache在slab初始化的时候已经生成好)。
    2. 通过slab_alloc从kmem_cache中分配一个slab对象，并返回。

    具体细节我们后续详细讨论slab_alloc的详细实现。

    kfree

    kfree最终会调用到___cache_free，具体我们再 slab free中详细讨论。

    初始化

    内核的大部分管理数据结构都是通过kmalloc分配内存的，那么slab本身结构的内存管理就出现了一个鸡与蛋的问题，slab数据结构所需内存远小于一整页的内存块，这些最适合kmalloc分配，而kmalloc只有在slab初始化完之后才能使用。


    所以就需要内核在启动时进行一些初始化操作，让内核在后续的数据结构能够找到对应的kmem_cache对象供kmalloc使用，而这一机制就是之前提到的kmalloc_caches。kernel启动时会先进行slab初始化，slab初始化时会先为kmalloc_caches分配内存，后续slab对象关联的kmem_cache可以从kmalloc_caches中获取，这样slab本身数据结构的内存分配就可以使用了，而这些内存会存在kmem_cache_boot这个缓存中。

    kmem_cache_init是初始化slab分配器的函数，其主要作用就是通过create_kmalloc_caches初始化kmalloc_caches。而kmalloc_caches数组中的kmem_cache对象本身则存储在kmem_cache_boot中，因为kmem_cache对象的大小是固定的，所以只需要一个kmem_cache就可以了，而该函数在start_kernel中被调用进行初始化，create_kmalloc_caches的实现如下：

    1. struct kmem_cache *
    2. kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init;
    3. EXPORT_SYMBOL(kmalloc_caches); /* [zone][index] */
    4.  
    5. /*
    6.  * Initialisation.  Called after the page allocator have been initialised and
    7.  * before smp_init().
    8.  */
    9. void __init kmem_cache_init(void)
    10. {
    11.     int I;
    12.  
    13.     kmem_cache = &kmem_cache_boot;
    14.     if (!IS_ENABLED(CONFIG_NUMA) || num_possible_nodes() == 1)
    15.         use_alien_caches = 0;
    16.     for (i = 0; i < NUM_INIT_LISTS; I++)
    17.         kmem_cache_node_init(&init_kmem_cache_node[i]); // 初始化kmem_cache_boot关联的kmem_cache_node，用于管理slab相关数据结构内存分配的实际存储，具体关联是在__kmem_cache_create中调用set_up_node(kmem_cache, CACHE_CACHE);以及如下代码的第5)会兜底关联一次。
    18.  
    19.     /*
    20.      * Fragmentation resistance on low memory - only use bigger
    21.      * page orders on machines with more than 32MB of memory if
    22.      * not overridden on the command line.
    23.      */
    24.     if (!slab_max_order_set && totalram_pages() > (32 << 20) >> PAGE_SHIFT)
    25.         slab_max_order = SLAB_MAX_ORDER_HI; // 设置slab每次申请的最大伙伴页帧
    26.  
    27.     /* Bootstrap is tricky, because several objects are allocated
    28.      * from caches that do not exist yet:
    29.      * 1) initialize the kmem_cache cache: it contains the struct
    30.      *    kmem_cache structures of all caches, except kmem_cache itself:
    31.      *    kmem_cache is statically allocated.
    32.      *    Initially an __init data area is used for the head array and the
    33.      *    kmem_cache_node structures, it's replaced with a kmalloc allocated
    34.      *    array at the end of the bootstrap.
    35.      * 2) Create the first kmalloc cache.
    36.      *    The struct kmem_cache for the new cache is allocated normally.
    37.      *    An __init data area is used for the head array.
    38.      * 3) Create the remaining kmalloc caches, with minimally sized
    39.      *    head arrays.
    40.      * 4) Replace the __init data head arrays for kmem_cache and the first
    41.      *    kmalloc cache with kmalloc allocated arrays.
    42.      * 5) Replace the __init data for kmem_cache_node for kmem_cache and
    43.      *    the other cache's with kmalloc allocated memory.
    44.      * 6) Resize the head arrays of the kmalloc caches to their final sizes.
    45.      */
    47.     /* 1) create the kmem_cache */
    48.  
    49.     /*
    50.      * struct kmem_cache size depends on nr_node_ids & nr_cpu_ids
    51.      */
    52.     create_boot_cache(kmem_cache, "kmem_cache",
    53.         offsetof(struct kmem_cache, node) +
    54.                   nr_node_ids * sizeof(struct kmem_cache_node *),
    55.                   SLAB_HWCACHE_ALIGN, 0, 0);
    56.     list_add(&kmem_cache->list, &slab_caches);
    57.     memcg_link_cache(kmem_cache);
    58.     slab_state = PARTIAL;
    59.  
    60.     /*
    61.      * Initialize the caches that provide memory for the  kmem_cache_node
    62.      * structures first.  Without this, further allocations will bug.
    63.      */
    64.     kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache(
    65.                 kmalloc_info[INDEX_NODE].name,
    66.                 kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS,
    67.                 0, kmalloc_size(INDEX_NODE));
    68.     slab_state = PARTIAL_NODE;
    69.     setup_kmalloc_cache_index_table();
    70.  
    71.     slab_early_init = 0;
    72.  
    73.     /* 5) Replace the bootstrap kmem_cache_node */
    74.     {
    75.         int nid;
    76.         for_each_online_node(nid) {
    77.             init_list(kmem_cache, &init_kmem_cache_node[CACHE_CACHE + nid], nid);
    78.             init_list(kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE],
    79.                       &init_kmem_cache_node[SIZE_NODE + nid], nid);
    80.         }
    81.     }
    82.     create_kmalloc_caches(ARCH_KMALLOC_FLAGS);
    83. }
    84.  
    85. /*
    86.  * Create the kmalloc array. Some of the regular kmalloc arrays
    87.  * may already have been created because they were needed to
    88.  * enable allocations for slab creation.
    89.  */
    90. void __init create_kmalloc_caches(slab_flags_t flags)
    91. {
    92.     int i, type;
    93.         // 从zone及size两个维度初始化kmalloc_caches
    94.     for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) {
    95.         for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) {
    96.             if (!kmalloc_caches[type][I])
    97.                 new_kmalloc_cache(i, type, flags);
    98.             /*
    99.              * Caches that are not of the two-to-the-power-of size.
    100.              * These have to be created immediately after the
    101.              * earlier power of two caches
    102.              */
    103.             if (KMALLOC_MIN_SIZE <= 32 && i == 6 &&
    104.                     !kmalloc_caches[type][1])
    105.                 new_kmalloc_cache(1, type, flags);
    106.             if (KMALLOC_MIN_SIZE <= 64 && i == 7 &&
    107.                     !kmalloc_caches[type][2])
    108.                 new_kmalloc_cache(2, type, flags);
    109.         }
    110.     }
    111.  
    112.     /* Kmalloc array is now usable */
    113.     slab_state = UP;
    114.  
    115. #ifdef CONFIG_ZONE_DMA
    116.     for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) {
    117.         struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i];
    118.  
    119.         if (s) {
    120.             unsigned int size = kmalloc_size(i);
    121.             const char *n = kmalloc_cache_name("dma-kmalloc", size);
    122.  
    123.             BUG_ON(!n);
    124.             kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(
    125.                 n, size, SLAB_CACHE_DMA | flags, 0, 0);
    126.         }
    127.     }
    128. #endif
    129. }
    130.  
    131. static void __init
    132. new_kmalloc_cache(int idx, int type, slab_flags_t flags)
    133. {
    134.     const char *name;
    135.  
    136.     if (type == KMALLOC_RECLAIM) {
    137.         flags |= SLAB_RECLAIM_ACCOUNT;
    138.         name = kmalloc_cache_name("kmalloc-rcl",
    139.                         kmalloc_info[idx].size);
    140.         BUG_ON(!name);
    141.     } else {
    142.         name = kmalloc_info[idx].name;
    143.     }
    144.  
    145.     kmalloc_caches[type][idx] = create_kmalloc_cache(name,
    146.                     kmalloc_info[idx].size, flags, 0,
    147.                     kmalloc_info[idx].size);
    148. }
    149.  
    150. /* internal cache of cache description objs */
    151. static struct kmem_cache kmem_cache_boot = {
    152.     .batchcount = 1,
    153.     .limit = BOOT_CPUCACHE_ENTRIES,
    154.     .shared = 1,
    155.     .size = sizeof(struct kmem_cache),
    156.     .name = "kmem_cache",
    157. };
    158.  
    159. /*
    160.  * 通过kmem_cache_zalloc分配对象内存(分配指定大小的内存，并填充为0字节)
    161.  * 然后通过create_boot_cache填充数据
    162.  */
    163. struct kmem_cache *__init create_kmalloc_cache(const char *name,
    164.         unsigned int size, slab_flags_t flags,
    165.         unsigned int useroffset, unsigned int usersize)
    166. {
    167.     //  kmem_cache == kmem_cache_boot
    168.     struct kmem_cache *s = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
    169.  
    170.     if (!s)
    171.         panic("Out of memory when creating slab %s\n", name);
    172.  
    173.     create_boot_cache(s, name, size, flags, useroffset, usersize); // 最终会调用到__kmem_cache_create初始化kmem_cache中的数据，这一块在后续的缓存创建中详细介绍。
    174.     list_add(&s->list, &slab_caches);
    175.     memcg_link_cache(s);
    176.     s->refcount = 1;
    177.     return s;
    178. }

    基本调用关系图如下：

    slab 初始化主要是为了初始化kmem_cache_boot及kmalloc_caches数组，方便后续内核核心代码使用kmalloc，而不需要额外管理slab内存数据结构，这样slab本身的数据结构内存申请就可以直接使用上kmalloc了。kmem_cache相关数据都是由kmem_cache_boot管理的，kmalloc调用时使用的kmem_cache都是在初始化时已经创建好的kmalloc_caches中的kmem_cache缓存对象，上述__kmem_cache_create就是创建kmem_cache对象的关键函数，这个我们在后续slab create中详细介绍。

    在介绍slab 初始化的时候有提到__kmem_cache_create。其实kmem_cache_create最终也是通过__kmem_cache_create来初始化kmem_cache相关变量，具体调用关系图如下：

    slab create的主要目的就是初始化一个kmem_cache缓存对象中的起始值，主要包括：batchcount，size，align，useroffset，usersize，flags，colour，num，gfporder等等。但是这里有特别需要注意的是：batchcount及limit在create的时候是设为1，且并没有分配相应的页帧来存储slab对象，这些都需要在slab alloc发现没有可用对象时进行grow分配。

    slab create根据需要被管理的对象size，计算kmem_cache_node中管理的伙伴页帧阶gfporder大小，我们把这样一个伙伴页帧叫做slab，通过slab中的页数以及对象size就可以计算出一个slab中能存储的对象数量num = ((PAGE_SIZE << gfporder) - head) / size 下取整。
    代码如下：

    1. /**
    2.  * __kmem_cache_create - Create a cache.
    3.  * @cachep: cache management descriptor
    4.  * @flags: SLAB flags
    5.  *
    6.  * Returns a ptr to the cache on success, NULL on failure.
    7.  * Cannot be called within a int, but can be interrupted.
    8.  * The @ctor is run when new pages are allocated by the cache.
    9.  *
    10.  * The flags are
    11.  *
    12.  * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5)
    13.  * to catch references to uninitialised memory.
    14.  *
    15.  * %SLAB_RED_ZONE - Insert `Red' zones around the allocated memory to check
    16.  * for buffer overruns.
    17.  *
    18.  * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware
    19.  * cacheline.  This can be beneficial if you're counting cycles as closely
    20.  * as davem.
    21.  */
    22. int __kmem_cache_create(struct kmem_cache *cachep, slab_flags_t flags)
    23. {
    24.     size_t ralign = BYTES_PER_WORD; // 按字对齐
    25.     gfp_t gfp;
    26.     int err;
    27.     unsigned int size = cachep->size;
    28.     ... // debug info
    29.     /*
    30.      * Check that size is in terms of words.  This is needed to avoid
    31.      * unaligned accesses for some archs when redzoning is used, and makes
    32.      * sure any on-slab bufctl's are also correctly aligned.
    33.      */
    34.     size = ALIGN(size, BYTES_PER_WORD);
    35.  
    36.     if (flags & SLAB_RED_ZONE) {
    37.         ralign = REDZONE_ALIGN;
    38.         /* If redzoning, ensure that the second redzone is suitably
    39.          * aligned, by adjusting the object size accordingly. */
    40.         // 危险区，在每个对象开始和结束处添加已知字节模式，方便程序员debug发现不正确的访问
    41.         size = ALIGN(size, REDZONE_ALIGN);
    42.     }
    43.  
    44.     /* 3) caller mandated alignment */
    45.     if (ralign < cachep->align) {
    46.         ralign = cachep->align;
    47.     }
    48.     /* disable debug if necessary */
    49.     if (ralign > __alignof__(unsigned long long))
    50.         flags &= ~(SLAB_RED_ZONE | SLAB_STORE_USER);
    51.     /*
    52.      * 4) Store it.
    53.      */
    54.     cachep->align = ralign;
    55.     cachep->colour_off = cache_line_size();
    56.     /* Offset must be a multiple of the alignment. */
    57.     if (cachep->colour_off < cachep->align)
    58.         cachep->colour_off = cachep->align;
    59.     if (slab_is_available())
    60.         gfp = GFP_KERNEL;
    61.     else
    62.         gfp = GFP_NOWAIT;
    63.     ... // debug info
    64.     kasan_cache_create(cachep, &size, &flags);
    65.  
    66.     size = ALIGN(size, cachep->align);
    67.     /*
    68.      * We should restrict the number of objects in a slab to implement
    69.      * byte sized index. Refer comment on SLAB_OBJ_MIN_SIZE definition.
    70.      */
    71.     if (FREELIST_BYTE_INDEX && size < SLAB_OBJ_MIN_SIZE)
    72.         size = ALIGN(SLAB_OBJ_MIN_SIZE, cachep->align);
    73.     ... // debug info
    74.     if (set_objfreelist_slab_cache(cachep, size, flags)) { // 计算gfporder and num
    75.         flags |= CFLGS_OBJFREELIST_SLAB;
    76.         goto done;
    77.     }
    78.     if (set_off_slab_cache(cachep, size, flags)) { // 计算gfporder and num
    79.         flags |= CFLGS_OFF_SLAB;
    80.         goto done;
    81.     }
    82.     if (set_on_slab_cache(cachep, size, flags)) // 计算gfporder and num
    83.         goto done;
    84.     return -E2BIG;
    85. done:
    86.     cachep->freelist_size = cachep->num * sizeof(freelist_idx_t);
    87.     cachep->flags = flags;
    88.     cachep->allocflags = __GFP_COMP;
    89.     if (flags & SLAB_CACHE_DMA)
    90.         cachep->allocflags |= GFP_DMA;
    91.     if (flags & SLAB_CACHE_DMA32)
    92.         cachep->allocflags |= GFP_DMA32;
    93.     if (flags & SLAB_RECLAIM_ACCOUNT)
    94.         cachep->allocflags |= __GFP_RECLAIMABLE;
    95.     cachep->size = size;
    96.     cachep->reciprocal_buffer_size = reciprocal_value(size);
    97. #if DEBUG
    98.     /*
    99.      * If we're going to use the generic kernel_map_pages()
    100.      * poisoning, then it's going to smash the contents of
    101.      * the redzone and userword anyhow, so switch them off.
    102.      */
    103.     if (IS_ENABLED(CONFIG_PAGE_POISONING) &&
    104.         (cachep->flags & SLAB_POISON) &&
    105.         is_debug_pagealloc_cache(cachep))
    106.         cachep->flags &= ~(SLAB_RED_ZONE | SLAB_STORE_USER);
    107. #endif
    108.     if (OFF_SLAB(cachep)) {
    109.         cachep->freelist_cache =
    110.             kmalloc_slab(cachep->freelist_size, 0u);
    111.     }
    112.     err = setup_cpu_cache(cachep, gfp);  // 初始化array_caches及 kmem_cache_node
    113.     if (err) {
    114.         __kmem_cache_release(cachep);
    115.         return err;
    116.     }
    117.     return 0;
    118. }
    119.  
    120. static int __ref setup_cpu_cache(struct kmem_cache *cachep, gfp_t gfp)
    121. {
    122.     if (slab_state >= FULL)
    123.         return enable_cpucache(cachep, gfp);
    124.     cachep->cpu_cache = alloc_kmem_cache_cpus(cachep, 1, 1);
    125.     if (!cachep->cpu_cache)
    126.         return 1;
    127.     /* init kmem_cache_node*/
    128.     if (slab_state == DOWN) {
    129.         /* Creation of first cache (kmem_cache). */
    130.         set_up_node(kmem_cache, CACHE_CACHE);
    131.     } else if (slab_state == PARTIAL) {
    132.         /* For kmem_cache_node */
    133.         set_up_node(cachep, SIZE_NODE);
    134.     } else {
    135.         int node;
    136.         for_each_online_node(node) {
    137.             cachep->node[node] = kmalloc_node(
    138.                 sizeof(struct kmem_cache_node), gfp, node);
    139.             BUG_ON(!cachep->node[node]);
    140.             kmem_cache_node_init(cachep->node[node]);
    141.         }
    142.     }
    143.     cachep->node[numa_mem_id()]->next_reap =
    144.             jiffies + REAPTIMEOUT_NODE +
    145.             ((unsigned long)cachep) % REAPTIMEOUT_NODE;
    146.      /* init array_caches*/
    147.     cpu_cache_get(cachep)->avail = 0;
    148.     cpu_cache_get(cachep)->limit = BOOT_CPUCACHE_ENTRIES;
    149.     cpu_cache_get(cachep)->batchcount = 1;
    150.     cpu_cache_get(cachep)->touched = 0;
    151.     cachep->batchcount = 1;
    152.     cachep->limit = BOOT_CPUCACHE_ENTRIES;
    153.     return 0;
    154. }

    这里需要特别单独说一下的是kmalloc_node，这是分配kmem_cache_node的关键函数，相关函数调用如下：

    相关代码如下：

    1. static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
    2. {
    3.   ... // slob 先不考虑
    4.     return __kmalloc_node(size, flags, node);
    5. }
    6. void *__kmalloc_node(size_t size, gfp_t flags, int node)
    7. {
    8.     return __do_kmalloc_node(size, flags, node, _RET_IP_);
    9. }
    10. static __always_inline void *
    11. __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller)
    12. {
    13.     struct kmem_cache *cachep;
    14.     void *ret;
    15.     if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
    16.         return NULL;
    17.     cachep = kmalloc_slab(size, flags);
    18.     if (unlikely(ZERO_OR_NULL_PTR(cachep)))
    19.         return cachep;
    20.     ret = kmem_cache_alloc_node_trace(cachep, flags, node, size);
    21.     ret = kasan_kmalloc(cachep, ret, size, flags);
    22.     return ret;
    23. }
    24. void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
    25.                   gfp_t flags,
    26.                   int nodeid,
    27.                   size_t size)
    28. {
    29.     void *ret;
    30.     ret = slab_alloc_node(cachep, flags, nodeid, _RET_IP_);
    31.     ret = kasan_kmalloc(cachep, ret, size, flags);
    32.     trace_kmalloc_node(_RET_IP_, ret,
    33.                size, cachep->size,
    34.                flags, nodeid);
    35.     return ret;
    36. }
    37.  
    38. static __always_inline void *
    39. slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
    40.            unsigned long caller)
    41. {
    42.     unsigned long save_flags;
    43.     void *ptr;
    44.     int slab_node = numa_mem_id();
    45.  
    46.   /* 参数及相关环境检查*/
    47.     flags &= gfp_allowed_mask;
    48.     cachep = slab_pre_alloc_hook(cachep, flags);
    49.     if (unlikely(!cachep))
    50.         return NULL;
    51.  
    52.     cache_alloc_debugcheck_before(cachep, flags);
    53.     local_irq_save(save_flags);
    54.  
    55.     if (nodeid == NUMA_NO_NODE)
    56.         nodeid = slab_node;
    57.  
    58.     if (unlikely(!get_node(cachep, nodeid))) {
    59.         /* Node not bootstrapped yet */
    60.         ptr = fallback_alloc(cachep, flags);
    61.         goto out;
    62.     }
    63.  
    64.     if (nodeid == slab_node) {
    65.         /*
    66.          * Use the locally cached objects if possible.
    67.          * However ____cache_alloc does not allow fallback
    68.          * to other nodes. It may fail while we still have
    69.          * objects on other nodes available.
    70.          */
    71.         ptr = ____cache_alloc(cachep, flags); // 从本地array_cache中直接分配对应的slab obj去存储kmem_cache_node
    72.         if (ptr)
    73.             goto out;
    74.     }
    75.     /* ___cache_alloc_node can fall back to other nodes */
    76.     ptr = ____cache_alloc_node(cachep, flags, nodeid);
    77.   out:
    78.     local_irq_restore(save_flags);
    79.     ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller);
    80.  
    81.     if (unlikely(flags & __GFP_ZERO) && ptr)
    82.         memset(ptr, 0, cachep->object_size);
    83.  
    84.     slab_post_alloc_hook(cachep, flags, 1, &ptr);
    85.     return ptr;
    86. }
    87.  
    88. /*
    89.  * A interface to enable slab creation on nodeid
    90.  */
    91. static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags,
    92.                 int nodeid)
    93. {
    94.     struct page *page;
    95.     struct kmem_cache_node *n;
    96.     void *obj = NULL;
    97.     void *list = NULL;
    98.  
    99.     VM_BUG_ON(nodeid < 0 || nodeid >= MAX_NUMNODES);
    100.     n = get_node(cachep, nodeid);
    101.     BUG_ON(!n);
    102.  
    103.     check_irq_off();
    104.     spin_lock(&n->list_lock);
    105.     page = get_first_slab(n, false);
    106.     if (!page)
    107.         goto must_grow;
    108.  
    109.     check_spinlock_acquired_node(cachep, nodeid);
    110.  
    111.     STATS_INC_NODEALLOCS(cachep);
    112.     STATS_INC_ACTIVE(cachep);
    113.     STATS_SET_HIGH(cachep);
    114.  
    115.     BUG_ON(page->active == cachep->num);
    116.  
    117.     obj = slab_get_obj(cachep, page);
    118.     n->free_objects--;
    119.  
    120.     fixup_slab_list(cachep, n, page, &list);
    121.  
    122.     spin_unlock(&n->list_lock);
    123.     fixup_objfreelist_debug(cachep, &list);
    124.     return obj;
    125.  
    126. must_grow:
    127.     spin_unlock(&n->list_lock);
    128.     page = cache_grow_begin(cachep, gfp_exact_node(flags), nodeid);
    129.     if (page) {
    130.         /* This slab isn't counted yet so don't update free_objects */
    131.         obj = slab_get_obj(cachep, page);
    132.     }
    133.     cache_grow_end(cachep, page);
    134.  
    135.     return obj ? obj : fallback_alloc(cachep, flags);
    136. }

    从上面的代码发现slab本身的管理对象也由slab对象直接管理。

    slab alloc

    slab alloc的基本过程如下，最终会调用到____cache_alloc，如果在array_cache中能够找到的话，就直接返回否则通过cache_alloc_refill重新填充array_cache中的空闲对象。

    相关代码如下：

    1. static inline void *____cache_alloc(struct kmem_cache *cachep, gfp_t flags)
    2. {
    3.     void *objp;
    4.     struct array_cache *ac;
    5.  
    6.     check_irq_off();
    7.  
    8.     ac = cpu_cache_get(cachep);
    9.     if (likely(ac->avail)) {
    10.         ac->touched = 1;
    11.         objp = ac->entry[--ac->avail];
    12.  
    13.         STATS_INC_ALLOCHIT(cachep);
    14.         goto out;
    15.     }
    16.  
    17.     STATS_INC_ALLOCMISS(cachep);
    18.     objp = cache_alloc_refill(cachep, flags);
    19.     /*
    20.      * the 'ac' may be updated by cache_alloc_refill(),
    21.      * and kmemleak_erase() requires its correct value.
    22.      */
    23.     ac = cpu_cache_get(cachep);
    24.  
    25. out:
    26.     /*
    27.      * To avoid a false negative, if an object that is in one of the
    28.      * per-CPU caches is leaked, we need to make sure kmemleak doesn't
    29.      * treat the array pointers as a reference to the object.
    30.      */
    31.     if (objp)
    32.         kmemleak_erase(&ac->entry[ac->avail]);
    33.     return objp;
    34. }
    35.  
    36. static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)
    37. {
    38.     int batchcount;
    39.     struct kmem_cache_node *n;
    40.     struct array_cache *ac, *shared;
    41.     int node;
    42.     void *list = NULL;
    43.     struct page *page;
    44.     check_irq_off();
    45.     node = numa_mem_id();
    46.     ac = cpu_cache_get(cachep);
    47.     batchcount = ac->batchcount;
    48.     if (!ac->touched && batchcount > BATCHREFILL_LIMIT) {
    49.         /*
    50.          * If there was little recent activity on this cache, then
    51.          * perform only a partial refill.  Otherwise we could generate
    52.          * refill bouncing.
    53.          */
    54.         batchcount = BATCHREFILL_LIMIT; // 16
    55.     }
    56.     n = get_node(cachep, node);
    57.     BUG_ON(ac->avail > 0 || !n);
    58.     shared = READ_ONCE(n->shared);
    59.     if (!n->free_objects && (!shared || !shared->avail))
    60.         goto direct_grow;
    61.  
    62.     spin_lock(&n->list_lock);
    63.     shared = READ_ONCE(n->shared);
    64.  
    65.     /* See if we can refill from the shared array */
    66.     if (shared && transfer_objects(ac, shared, batchcount)) {
    67.         shared->touched = 1;
    68.         goto alloc_done;
    69.     }
    70.     while (batchcount > 0) {
    71.         /* Get slab alloc is to come from. */
    72.         page = get_first_slab(n, false);
    73.         if (!page)
    74.             goto must_grow;
    75.         check_spinlock_acquired(cachep);
    76.         batchcount = alloc_block(cachep, ac, page, batchcount);
    77.         fixup_slab_list(cachep, n, page, &list);
    78.     }
    79. must_grow:
    80.     n->free_objects -= ac->avail;
    81. alloc_done:
    82.     spin_unlock(&n->list_lock);
    83.     fixup_objfreelist_debug(cachep, &list);
    84. direct_grow:
    85.     if (unlikely(!ac->avail)) {
    86.         /* Check if we can use obj in pfmemalloc slab */
    87.         if (sk_memalloc_socks()) {
    88.             void *obj = cache_alloc_pfmemalloc(cachep, n, flags);
    89.             if (obj)
    90.                 return obj;
    91.         }
    92.         page = cache_grow_begin(cachep, gfp_exact_node(flags), node); // 分配一个gfporder伙伴页帧，并初始化伙伴页帧中的freelist
    93.         /*
    94.          * cache_grow_begin() can reenable interrupts,
    95.          * then ac could change.
    96.          */
    97.         ac = cpu_cache_get(cachep);
    98.         if (!ac->avail && page)
    99.             alloc_block(cachep, ac, page, batchcount);
    100.         cache_grow_end(cachep, page);// 将新申请的伙伴页帧加入到kmem_cache_node中的free或者partial或者full列表中
    101.         if (!ac->avail)
    102.             return NULL;
    103.     }
    104.     ac->touched = 1;
    105.     return ac->entry[--ac->avail];
    106. }
    107.  
    108. static struct page *get_first_slab(struct kmem_cache_node *n, bool pfmemalloc)
    109. {
    110.     struct page *page;
    111.     assert_spin_locked(&n->list_lock);
    112.     page = list_first_entry_or_null(&n->slabs_partial, struct page, lru); // 尝试获取部分空闲的page
    113.     if (!page) {
    114.         n->free_touched = 1;
    115.         page = list_first_entry_or_null(&n->slabs_free, struct page, lru); // 尝试获取全部空闲的page
    116.         if (page)
    117.             n->free_slabs--;
    118.     }
    119.     if (sk_memalloc_socks()) // 这种优化方案目前不知道是怎么搞的
    120.         page = get_valid_first_slab(n, page, pfmemalloc);
    121.     return page;
    122. }
    123.  
    124. /*
    125.  * Grow (by 1) the number of slabs within a cache.  This is called by
    126.  * kmem_cache_alloc() when there are no active objs left in a cache.
    127.  */
    128. static struct page *cache_grow_begin(struct kmem_cache *cachep,
    129.                 gfp_t flags, int nodeid)
    130. {
    131.     ...
    132.     /*
    133.      * Be lazy and only check for valid flags here,  keeping it out of the
    134.      * critical path in kmem_cache_alloc().
    135.      */
    136.    ...
    137.     /*
    138.      * Get mem for the objs.  Attempt to allocate a physical page from
    139.      * 'nodeid'.
    140.      */
    141.     page = kmem_getpages(cachep, local_flags, nodeid); // 在对应的内存节点通过伙伴系统分配一个gfporder的连续页
    142.     if (!page)
    143.         goto failed;
    144.     page_node = page_to_nid(page);
    145.     n = get_node(cachep, page_node);
    146.     /* Get colour for the slab, and cal the next value. */
    147.     ... // 着色
    148.     /*
    149.      * Call kasan_poison_slab() before calling alloc_slabmgmt(), so
    150.      * page_address() in the latter returns a non-tagged pointer,
    151.      * as it should be for slab pages.
    152.      */
    153.     kasan_poison_slab(page); // 毒化，在建立和释放slab时，易于感知未授权访问
    154.  
    155.     /* Get slab management. */
    156.     freelist = alloc_slabmgmt(cachep, page, offset,
    157.             local_flags & ~GFP_CONSTRAINT_MASK, page_node); // 获取第一个空闲对象地址，一般来说是在指定offset(着色偏移)之后的地址。
    158.     if (OFF_SLAB(cachep) && !freelist)
    159.         goto opps1;
    160.     slab_map_pages(cachep, page, freelist); // 初始化page slab信息
    161.     cache_init_objs(cachep, page);  // 将页帧中所有slab对象起始地址随机打散到freelist列表中， 
    162.     if (gfpflags_allow_blocking(local_flags))
    163.         local_irq_disable();
    164.     return page;
    165. opps1:
    166.     kmem_freepages(cachep, page);
    167. failed:
    168.     if (gfpflags_allow_blocking(local_flags))
    169.         local_irq_disable();
    170.     return NULL;
    171. }
    172.  
    173. static void cache_init_objs(struct kmem_cache *cachep,
    174.                 struct page *page)
    175. {
    176.     int I;
    177.     void *objp;
    178.     bool shuffled;
    179.  
    180.     cache_init_objs_debug(cachep, page);
    181.  
    182.     /* Try to randomize the freelist if enabled */
    183.     shuffled = shuffle_freelist(cachep, page);
    184.  
    185.     if (!shuffled && OBJFREELIST_SLAB(cachep)) {
    186.         page->freelist = index_to_obj(cachep, page, cachep->num - 1) +
    187.                         obj_offset(cachep);
    188.     }
    189.  
    190.     for (i = 0; i < cachep->num; i++) {
    191.         objp = index_to_obj(cachep, page, i);
    192.         objp = kasan_init_slab_obj(cachep, objp);
    193.  
    194.         /* constructor could break poison info */
    195.         if (DEBUG == 0 && cachep->ctor) {
    196.             kasan_unpoison_object_data(cachep, objp);
    197.             cachep->ctor(objp);
    198.             kasan_poison_object_data(cachep, objp);
    199.         }
    200.  
    201.         if (!shuffled)
    202.             set_free_obj(page, i, i);
    203.     }
    204. }
    205.  
    206. /*
    207.  * Shuffle the freelist initialization state based on pre-computed lists.
    208.  * return true if the list was successfully shuffled, false otherwise.
    209.  */
    210. static bool shuffle_freelist(struct kmem_cache *cachep, struct page *page)
    211. {
    212.     unsigned int objfreelist = 0, i, rand, count = cachep->num;
    213.     union freelist_init_state state;
    214.     bool precomputed;
    215.  
    216.     if (count < 2)
    217.         return false;
    218.  
    219.     precomputed = freelist_state_initialize(&state, cachep, count);
    220.  
    221.     /* Take a random entry as the objfreelist */
    222.     if (OBJFREELIST_SLAB(cachep)) {
    223.         if (!precomputed)
    224.             objfreelist = count - 1;
    225.         else
    226.             objfreelist = next_random_slot(&state);
    227.         page->freelist = index_to_obj(cachep, page, objfreelist) +
    228.                         obj_offset(cachep);
    229.         count--;
    230.     }
    231.  
    232.     /*
    233.      * On early boot, generate the list dynamically.
    234.      * Later use a pre-computed list for speed.
    235.      */
    236.     if (!precomputed) {
    237.         for (i = 0; i < count; I++)
    238.             set_free_obj(page, i, i);
    239.  
    240.         /* Fisher-Yates shuffle */
    241.         for (i = count - 1; i > 0; i--) {
    242.             rand = prandom_u32_state(&state.rnd_state);
    243.             rand %= (i + 1);
    244.             swap_free_obj(page, i, rand);
    245.         }
    246.     } else {
    247.         for (i = 0; i < count; I++)
    248.             set_free_obj(page, i, next_random_slot(&state));
    249.     }
    250.  
    251.     if (OBJFREELIST_SLAB(cachep))
    252.         set_free_obj(page, cachep->num - 1, objfreelist);
    253.  
    254.     return true;
    255. }

    最终slab对象分配情况如下图：（起始freelist在实际场景中是随意的交叉错乱指向任意一块slab对象，不会像下面这样整齐）

    结合之前的slab data struct图，应该基本就对slab alloc有个比较清晰的理解了。

    slab free

    kmem_cache_free函数时释放slab对象的一个主要入口，函数调用图如下：

    slab free

    相关代码如下：

    1. void ___cache_free(struct kmem_cache *cachep, void *objp,
    2.         unsigned long caller)
    3. {
    4.     struct array_cache *ac = cpu_cache_get(cachep);
    5.  
    6.     check_irq_off();
    7.     kmemleak_free_recursive(objp, cachep->flags);
    8.     objp = cache_free_debugcheck(cachep, objp, caller);
    9.  
    10.     /*
    11.      * Skip calling cache_free_alien() when the platform is not numa.
    12.      * This will avoid cache misses that happen while accessing slabp (which
    13.      * is per page memory  reference) to get nodeid. Instead use a global
    14.      * variable to skip the call, which is mostly likely to be present in
    15.      * the cache.
    16.      */
    17.     if (nr_online_nodes > 1 && cache_free_alien(cachep, objp))
    18.         return;
    19.  
    20.     if (ac->avail < ac->limit) {
    21.         STATS_INC_FREEHIT(cachep);
    22.     } else {
    23.         STATS_INC_FREEMISS(cachep);
    24.         cache_flusharray(cachep, ac); // 超出了array_caches limit，需要收缩
    25.     }
    26.     if (sk_memalloc_socks()) {
    27.         struct page *page = virt_to_head_page(objp);
    28.         if (unlikely(PageSlabPfmemalloc(page))) {
    29.             cache_free_pfmemalloc(cachep, page, objp);
    30.             return;
    31.         }
    32.     }
    33.     ac->entry[ac->avail++] = objp;
    34. }
    35.  
    36. static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac)
    37. {
    38.     int batchcount;
    39.     struct kmem_cache_node *n;
    40.     int node = numa_mem_id();
    41.     LIST_HEAD(list);
    42.  
    43.     batchcount = ac->batchcount;
    44.  
    45.     check_irq_off();
    46.     n = get_node(cachep, node);
    47.     spin_lock(&n->list_lock);
    48.     if (n->shared) { // 优先放回shared区
    49.         struct array_cache *shared_array = n->shared;
    50.         int max = shared_array->limit - shared_array->avail;
    51.         if (max) {
    52.             if (batchcount > max)
    53.                 batchcount = max;
    54.             memcpy(&(shared_array->entry[shared_array->avail]),
    55.                    ac->entry, sizeof(void *) * batchcount);
    56.             shared_array->avail += batchcount;
    57.             goto free_done;
    58.         }
    59.     }
    60.     free_block(cachep, ac->entry, batchcount, node, &list); // 否则只能放回到kmem_cache_node的页帧中，并将可被回收的pages放到list链表中
    61. free_done:
    62. #if STATS
    63.     {
    64.         int i = 0;
    65.         struct page *page;
    66.  
    67.         list_for_each_entry(page, &n->slabs_free, lru) {
    68.             BUG_ON(page->active);
    69.  
    70.             I++;
    71.         }
    72.         STATS_SET_FREEABLE(cachep, i);
    73.     }
    74. #endif
    75.     spin_unlock(&n->list_lock);
    76.     slabs_destroy(cachep, &list); // 将可回收的页帧释放回伙伴系统
    77.     ac->avail -= batchcount;
    78.     memmove(ac->entry, &(ac->entry[batchcount]), sizeof(void *)*ac->avail);
    79. }

    总结

    简而言之，slab缓存的基本思想如下：

    内核使用slab分配器时，主要包括两个步骤：

    1. 通过kmem_cache_create创建一个缓存。
    2. 通过kmem_cache_alloc在指定缓存中申请一个slab对象。

    细分描述如下：

    1. 通过kmem_cache_create(size, flags)函数创建一个缓存对象，根据size计算缓存的gfporder，从而可以知道每次新分配管理页帧的数量，也就可以计算一个kmem_cache_node页帧中能管理的slab对象数目num。
    2. 当通过kmem_cache_create(*cachep, flags)函数申请slab对象分配时，首先查找cachep->array_caches[curr_cpu]中是否有可用空闲对象，有则直接返回，没有则执行步骤3。
    3. 从kmem_cache_node中一次导入batchcount数量的空闲对象到array_caches，导入规则是，先选择部分空闲页帧，然后是全部空闲页帧，将内存中对象装载到array_cache中，当没有可用页帧时执行grow，跳入步骤4，否则跳入步骤6。
    4. 通过伙伴系统申请2^gfporder数量的页，并初始化slab头部信息及空闲列表信息，就每个空闲对象的起始地址随机打散到page->freelist数组中，这样每次分配时指向的地址会比较随机。并将该页帧伙伴放入kmem_cache_node->slab_free中。
    5. 继续像步骤3一样继续装载空闲对象到array_cache，直至batchcount为0后跳入步骤6。
      6.返回array_cache->entry[--array_cache->avail];

    具体可以结合slab data struct图一起看，应该会有一个比较清晰的整体感观。

    slab coloring：

    在深入理解linux内核架构原文中，是这么描述的:如果数据存储在伙伴系统直接提供的页中，那么其地址总是出现在2的幂次的整数倍附近。这对CPU高速缓存的利用有负面影响，由于这种地址分布，使得某些缓存行过度使用，而其他的则几乎为空。通过slab着色(slab coloring)，slab分配器能够均匀的分布对象，以实现均匀的缓存利用，在看了slab coloring的实现原理后我百思不得其解，为啥通过对页内slab对象加一个偏移就可以让缓存命中均匀了呢，通过翻阅大量资料发现，其实这个slab coloring好像并没有什么卵用。

    要理解slab着色并不能让缓存利用更均匀，首先得对cpu cache有一定了解：cpu cache将cache划分成固定大小的cache line，一般来说一个cache line的大小为64Byte，然后将cache line与内存地址映射。cpu cache目前主要有三种映射：1、直接映射；2、全相连映射；3、组相连映射。缓存行由三个部分组成 valid，tag，data；其中valid表示该缓存行中数据是否有效，tag用于判断对应cach行存储的确实是目标地址，data则是缓存行具体的64Byte数据。

    我们以32位物理内存地址，1MB cache，64Byte cache line系统为例：
    那么该系统缓存地址有20位(2^20 = 1MB)，其中低6位用于定位缓存行内数据，高14位用于定位缓存行，而我们的物理内存地址有32位，那么同一缓存行中可能存储2^12种不同地址，所以就需要一个12位的TAG来确定缓存中存储的确实是指定物理地址的数据。比如0x000XXX和0x001XXX都会命中同一缓存行，而通过TAG中存储的值来确认存储的是0x000XXX还是0x001XXX的数据。

    具体过程如下：

    而缓存行的定位跟缓存与物理地址映射有关：

    直接映射

    直接映射中cache地址与内存地址是固定好的映射关系；因此可能会存在某些行过热的情况出现(某些倍数地址临时访问频繁，这种情况在大数据处理中很常见，hash shuffle的时候有时候很容易出现长尾)。

    全相连映射

    全相连映射中，每一个内存块可以随意的映射到任意一个cache line，由于可以随意映射，所以每次访问的时候要匹配每个cache line的tag来确认是否是对应的物理地址。

    组相连映射

    组相连映射是直接映射和全相连映射的折中，组间直接映射，组内全相连映射，这样一定程度上可以缓解热点问题，又能避免对比太多TAG。

    经过上面的分析，对于CPU hardware cache有了大致的了解。那么slab coloring这种偏移页内一个cache line的做法对于避免缓存行有什么帮助呢？看上去并没有什么太多用，看下面的分析：

    比如一个slab A分配的page起始地址为0X00000000，另一个slab B分配的page起始地址为0X00002000，cache的大小为1MB，每个缓存行大小为64Byte，一共有16384行。那么slab A对应的缓存行为0~63行，而slab B对应的缓存行为128~191行，他们本来就不会命中到同一缓存行，你页内再怎么偏移也没有任何关系。即使如果两个slab对应的page会命中到同一缓存行，页内的偏移也看上去不会有什么不同的改善，是不是？反而组相连可以比较好的解决相同命中时的冲突问题。

    但是仔细想一想kmem_cache_node中的slab页帧结构你会发现，如果没有着色偏移的话，那么每次对应的缓存行一般都是从一个固定值开始的，假如我们忽略slab head的大小，每次slab对象都从页的起始位置开始分配，由于slab对象与cache line 一般来说是不等的，那么每次映射的缓存行一般是0 ~ 61或者0 ~ 62这种，所以62，63这种缓存行就会有较低访问概率，所以就需要这个着色偏移，让首尾缓存访问的概率趋于相同

  • 相关阅读:
    sylar高性能服务器-日志(P1-P6)代码解析+调试分析
    可视化 | （三）Edward Tufted基本设计准则
    监控到底要哪些指标？
    【C++模块实现】| 【07】对于互斥、自旋锁、条件变量、信号量简介及封装
    【linux命令讲解大全】113.网络接口和系统设备监测工具ifstat和iostat的使用
    华为常用命令
    《算法通关村—最大小栈问题解析》
    大数据调度最佳实践 | 从Airflow迁移到Apache DolphinScheduler
    String面试总结
    【语义分割 01】Open MMLab介绍
  • 原文地址：https://blog.csdn.net/m0_74282605/article/details/127819805