preallocate 기능의 정확성 검토 및 정리.

[jhpark816]

slab preallocate 기능의 정확성을 검토하고 문제가 있으면 수정한다.
최신 memcached 코드도 참고 바람.

[6unYoung]

실행 흐름 비교

slab preallocate 처리에 대한 순서도만 나타낸 것입니다.

1. 원본 memcached

graph TD
    A["main()"] --> B{enable_large_pages:<br>preallocate 가능 여부};
    B -- yes --> C["preallocate = true"];
    C --> D["slab_init()"];
    B -- no --> D;
    D --> E{"prealloc == true"};
    E -- yes --> F["slab_preallocate()"];

Loading

2. arcus memcached
   default_initialize()에서 config를 parsing 후 slabs_init()에 전달하고 있습니다.

graph TD
    A["main()"] --> B{enable_large_pages:<br>preallocate 가능 여부};
    B -- yes --> C[engine_config에<br>preallocate=true 추가];
    C --> D["init_engine()"];
    B -- no --> D;
    D --> E["default_initialize()"];
    E --> F["initialize_configuration()<br>parse_config()"];
    F --> H["slabs_init()"];
    H --> I{"조건*"};
    I -- yes --> J["slabs_preallocate()"];

Loading

문제점

slabs.c:slab_init()에서 slabs_preallocate를 호출하는 조건*이 옳지 않아 보입니다. 현재는 pre_alloc이 NULL 또는 정수인 경우에 preallocate를 하게 되어있습니다. 하지만 해당 환경변수를 putenv()하는 구현부가 없어 항상 NULL이므로 옵션과 상관없이 preallocate를 수행하게 됩니다.
또한, 옵션을 줬을 때 환경변수를 할당하는 코드를 작성하더라도 환경변수가 없는 경우(옵션을 주지 않은 경우) preallocate를 수행하지 않는 것이 일반적인 작동이라고 생각합니다.
따라서 해당 조건문은 바뀌는 게 맞다면 원본 memcached와 동일하게 slabs_init()의 인자인 const bool prealloc를 사용하는 게 변수의 본래 의도에도 부합해 자연스러워 보입니다.

 #ifndef DONT_PREALLOC_SLABS
     {
-        char *pre_alloc = getenv("T_MEMD_SLABS_ALLOC");
-
-        if (pre_alloc == NULL || atoi(pre_alloc) != 0) {
-            slabs_preallocate(power_largest);
+       if (prealloc) {
+            slabs_preallocate(slabsp->power_largest);
         }
     }
 #endif

추가 수정점

위의 조건문 외에도 다음 코드들을 수정했습니다.

// memcached.c:15587
-     //bool preallocate = false;
+    bool __attribute__((unused)) preallocate = false;

// memcached.c:
         case 'L' :15800
             if (enable_large_pages() == 0) {
-                //preallocate = true;
+                preallocate = true;
                 old_opts += sprintf(old_opts, "preallocate=true;");
             }
             break;

// engines/default/slabs.c:42
-#define DONT_PREALLOC_SLABS
+//#define DONT_PREALLOC_SLABS

slab preallocate를 사용하는 경우 slabs_preallocate의 에러 처리를 해줘야 할 것 같습니다. 에러 처리 방식은 원본 memcached의 코드를 차용했습니다.

// engines/default/slabs.c:slabs_preallocate
...
     for (i = POWER_SMALLEST; i <= POWER_LARGEST; i++) {
         if (++prealloc > maxslabs)
             return;
-        do_slabs_newslab(i);
+        if (do_slabs_newslab(i) == 0) {
+            fprintf(stderr, "Error while preallocating slab memory!\n"
+                "If using -L or other prealloc options, max memory must be "
+                "at least %d megabytes.\n", maxslabs);
+            exit(1);
+        }
     }
...

[6unYoung]

테스트 결과

코드 수정 후 테스트한 결과입니다.

1. 원본 memcached, preallocate X

$ stats slabs
STAT active_slabs 0
STAT total_malloced 0
END

$ set 01 0 0 2
01
STORED

$ stats slabs
STAT 1:chunk_size 96
STAT 1:chunks_per_page 10922
STAT 1:total_pages 1
STAT 1:total_chunks 10922
STAT 1:used_chunks 1
STAT 1:free_chunks 10921
STAT 1:free_chunks_end 0
STAT 1:get_hits 0
STAT 1:cmd_set 1
STAT 1:delete_hits 0
STAT 1:incr_hits 0
STAT 1:decr_hits 0
STAT 1:cas_hits 0
STAT 1:cas_badval 0
STAT 1:touch_hits 0
STAT active_slabs 1
STAT total_malloced 1048576
END

2. 원본 memcached, preallocate O

$ stats slabs
STAT 1:chunk_size 96
STAT 1:chunks_per_page 10922
STAT 1:total_pages 1
STAT 1:total_chunks 10922
STAT 1:used_chunks 0
STAT 1:free_chunks 10922
STAT 1:free_chunks_end 0
STAT 1:get_hits 0
STAT 1:cmd_set 0
STAT 1:delete_hits 0
STAT 1:incr_hits 0
STAT 1:decr_hits 0
STAT 1:cas_hits 0
STAT 1:cas_badval 0
STAT 1:touch_hits 0
STAT 2:chunk_size 120
STAT 2:chunks_per_page 8738
STAT 2:total_pages 1
STAT 2:total_chunks 8738
STAT 2:used_chunks 0
STAT 2:free_chunks 8738
STAT 2:free_chunks_end 0
STAT 2:get_hits 0
STAT 2:cmd_set 0
STAT 2:delete_hits 0
STAT 2:incr_hits 0
STAT 2:decr_hits 0
STAT 2:cas_hits 0
STAT 2:cas_badval 0
STAT 2:touch_hits 0
STAT 3:chunk_size 152
STAT 3:chunks_per_page 6898
STAT 3:total_pages 1
STAT 3:total_chunks 6898
STAT 3:used_chunks 0
STAT 3:free_chunks 6898
STAT 3:free_chunks_end 0
STAT 3:get_hits 0
STAT 3:cmd_set 0
STAT 3:delete_hits 0
STAT 3:incr_hits 0
STAT 3:decr_hits 0
STAT 3:cas_hits 0
STAT 3:cas_badval 0
STAT 3:touch_hits 0
STAT 4:chunk_size 192
STAT 4:chunks_per_page 5461
STAT 4:total_pages 1
STAT 4:total_chunks 5461
STAT 4:used_chunks 0
STAT 4:free_chunks 5461
STAT 4:free_chunks_end 0
STAT 4:get_hits 0
STAT 4:cmd_set 0
STAT 4:delete_hits 0
STAT 4:incr_hits 0
STAT 4:decr_hits 0
STAT 4:cas_hits 0
STAT 4:cas_badval 0
STAT 4:touch_hits 0
STAT 5:chunk_size 240
STAT 5:chunks_per_page 4369
STAT 5:total_pages 1
STAT 5:total_chunks 4369
STAT 5:used_chunks 0
STAT 5:free_chunks 4369
STAT 5:free_chunks_end 0
STAT 5:get_hits 0
STAT 5:cmd_set 0
STAT 5:delete_hits 0
STAT 5:incr_hits 0
STAT 5:decr_hits 0
STAT 5:cas_hits 0
STAT 5:cas_badval 0
STAT 5:touch_hits 0
STAT 6:chunk_size 304
STAT 6:chunks_per_page 3449
STAT 6:total_pages 1
STAT 6:total_chunks 3449
STAT 6:used_chunks 0
STAT 6:free_chunks 3449
STAT 6:free_chunks_end 0
STAT 6:get_hits 0
STAT 6:cmd_set 0
STAT 6:delete_hits 0
STAT 6:incr_hits 0
STAT 6:decr_hits 0
STAT 6:cas_hits 0
STAT 6:cas_badval 0
STAT 6:touch_hits 0
STAT 7:chunk_size 384
STAT 7:chunks_per_page 2730
STAT 7:total_pages 1
STAT 7:total_chunks 2730
STAT 7:used_chunks 0
STAT 7:free_chunks 2730
STAT 7:free_chunks_end 0
STAT 7:get_hits 0
STAT 7:cmd_set 0
STAT 7:delete_hits 0
STAT 7:incr_hits 0
STAT 7:decr_hits 0
STAT 7:cas_hits 0
STAT 7:cas_badval 0
STAT 7:touch_hits 0
STAT 8:chunk_size 480
STAT 8:chunks_per_page 2184
STAT 8:total_pages 1
STAT 8:total_chunks 2184
STAT 8:used_chunks 0
STAT 8:free_chunks 2184
STAT 8:free_chunks_end 0
STAT 8:get_hits 0
STAT 8:cmd_set 0
STAT 8:delete_hits 0
STAT 8:incr_hits 0
STAT 8:decr_hits 0
STAT 8:cas_hits 0
STAT 8:cas_badval 0
STAT 8:touch_hits 0
STAT 9:chunk_size 600
STAT 9:chunks_per_page 1747
STAT 9:total_pages 1
STAT 9:total_chunks 1747
STAT 9:used_chunks 0
STAT 9:free_chunks 1747
STAT 9:free_chunks_end 0
STAT 9:get_hits 0
STAT 9:cmd_set 0
STAT 9:delete_hits 0
STAT 9:incr_hits 0
STAT 9:decr_hits 0
STAT 9:cas_hits 0
STAT 9:cas_badval 0
STAT 9:touch_hits 0
STAT 10:chunk_size 752
STAT 10:chunks_per_page 1394
STAT 10:total_pages 1
STAT 10:total_chunks 1394
STAT 10:used_chunks 0
STAT 10:free_chunks 1394
STAT 10:free_chunks_end 0
STAT 10:get_hits 0
STAT 10:cmd_set 0
STAT 10:delete_hits 0
STAT 10:incr_hits 0
STAT 10:decr_hits 0
STAT 10:cas_hits 0
STAT 10:cas_badval 0
STAT 10:touch_hits 0
STAT 11:chunk_size 944
STAT 11:chunks_per_page 1110
STAT 11:total_pages 1
STAT 11:total_chunks 1110
STAT 11:used_chunks 0
STAT 11:free_chunks 1110
STAT 11:free_chunks_end 0
STAT 11:get_hits 0
STAT 11:cmd_set 0
STAT 11:delete_hits 0
STAT 11:incr_hits 0
STAT 11:decr_hits 0
STAT 11:cas_hits 0
STAT 11:cas_badval 0
STAT 11:touch_hits 0
STAT 12:chunk_size 1184
STAT 12:chunks_per_page 885
STAT 12:total_pages 1
STAT 12:total_chunks 885
STAT 12:used_chunks 0
STAT 12:free_chunks 885
STAT 12:free_chunks_end 0
STAT 12:get_hits 0
STAT 12:cmd_set 0
STAT 12:delete_hits 0
STAT 12:incr_hits 0
STAT 12:decr_hits 0
STAT 12:cas_hits 0
STAT 12:cas_badval 0
STAT 12:touch_hits 0
STAT 13:chunk_size 1480
STAT 13:chunks_per_page 708
STAT 13:total_pages 1
STAT 13:total_chunks 708
STAT 13:used_chunks 0
STAT 13:free_chunks 708
STAT 13:free_chunks_end 0
STAT 13:get_hits 0
STAT 13:cmd_set 0
STAT 13:delete_hits 0
STAT 13:incr_hits 0
STAT 13:decr_hits 0
STAT 13:cas_hits 0
STAT 13:cas_badval 0
STAT 13:touch_hits 0
STAT 14:chunk_size 1856
STAT 14:chunks_per_page 564
STAT 14:total_pages 1
STAT 14:total_chunks 564
STAT 14:used_chunks 0
STAT 14:free_chunks 564
STAT 14:free_chunks_end 0
STAT 14:get_hits 0
STAT 14:cmd_set 0
STAT 14:delete_hits 0
STAT 14:incr_hits 0
STAT 14:decr_hits 0
STAT 14:cas_hits 0
STAT 14:cas_badval 0
STAT 14:touch_hits 0
STAT 15:chunk_size 2320
STAT 15:chunks_per_page 451
STAT 15:total_pages 1
STAT 15:total_chunks 451
STAT 15:used_chunks 0
STAT 15:free_chunks 451
STAT 15:free_chunks_end 0
STAT 15:get_hits 0
STAT 15:cmd_set 0
STAT 15:delete_hits 0
STAT 15:incr_hits 0
STAT 15:decr_hits 0
STAT 15:cas_hits 0
STAT 15:cas_badval 0
STAT 15:touch_hits 0
STAT 16:chunk_size 2904
STAT 16:chunks_per_page 361
STAT 16:total_pages 1
STAT 16:total_chunks 361
STAT 16:used_chunks 0
STAT 16:free_chunks 361
STAT 16:free_chunks_end 0
STAT 16:get_hits 0
STAT 16:cmd_set 0
STAT 16:delete_hits 0
STAT 16:incr_hits 0
STAT 16:decr_hits 0
STAT 16:cas_hits 0
STAT 16:cas_badval 0
STAT 16:touch_hits 0
STAT 17:chunk_size 3632
STAT 17:chunks_per_page 288
STAT 17:total_pages 1
STAT 17:total_chunks 288
STAT 17:used_chunks 0
STAT 17:free_chunks 288
STAT 17:free_chunks_end 0
STAT 17:get_hits 0
STAT 17:cmd_set 0
STAT 17:delete_hits 0
STAT 17:incr_hits 0
STAT 17:decr_hits 0
STAT 17:cas_hits 0
STAT 17:cas_badval 0
STAT 17:touch_hits 0
STAT 18:chunk_size 4544
STAT 18:chunks_per_page 230
STAT 18:total_pages 1
STAT 18:total_chunks 230
STAT 18:used_chunks 0
STAT 18:free_chunks 230
STAT 18:free_chunks_end 0
STAT 18:get_hits 0
STAT 18:cmd_set 0
STAT 18:delete_hits 0
STAT 18:incr_hits 0
STAT 18:decr_hits 0
STAT 18:cas_hits 0
STAT 18:cas_badval 0
STAT 18:touch_hits 0
STAT 19:chunk_size 5680
STAT 19:chunks_per_page 184
STAT 19:total_pages 1
STAT 19:total_chunks 184
STAT 19:used_chunks 0
STAT 19:free_chunks 184
STAT 19:free_chunks_end 0
STAT 19:get_hits 0
STAT 19:cmd_set 0
STAT 19:delete_hits 0
STAT 19:incr_hits 0
STAT 19:decr_hits 0
STAT 19:cas_hits 0
STAT 19:cas_badval 0
STAT 19:touch_hits 0
STAT 20:chunk_size 7104
STAT 20:chunks_per_page 147
STAT 20:total_pages 1
STAT 20:total_chunks 147
STAT 20:used_chunks 0
STAT 20:free_chunks 147
STAT 20:free_chunks_end 0
STAT 20:get_hits 0
STAT 20:cmd_set 0
STAT 20:delete_hits 0
STAT 20:incr_hits 0
STAT 20:decr_hits 0
STAT 20:cas_hits 0
STAT 20:cas_badval 0
STAT 20:touch_hits 0
STAT 21:chunk_size 8880
STAT 21:chunks_per_page 118
STAT 21:total_pages 1
STAT 21:total_chunks 118
STAT 21:used_chunks 0
STAT 21:free_chunks 118
STAT 21:free_chunks_end 0
STAT 21:get_hits 0
STAT 21:cmd_set 0
STAT 21:delete_hits 0
STAT 21:incr_hits 0
STAT 21:decr_hits 0
STAT 21:cas_hits 0
STAT 21:cas_badval 0
STAT 21:touch_hits 0
STAT 22:chunk_size 11104
STAT 22:chunks_per_page 94
STAT 22:total_pages 1
STAT 22:total_chunks 94
STAT 22:used_chunks 0
STAT 22:free_chunks 94
STAT 22:free_chunks_end 0
STAT 22:get_hits 0
STAT 22:cmd_set 0
STAT 22:delete_hits 0
STAT 22:incr_hits 0
STAT 22:decr_hits 0
STAT 22:cas_hits 0
STAT 22:cas_badval 0
STAT 22:touch_hits 0
STAT 23:chunk_size 13880
STAT 23:chunks_per_page 75
STAT 23:total_pages 1
STAT 23:total_chunks 75
STAT 23:used_chunks 0
STAT 23:free_chunks 75
STAT 23:free_chunks_end 0
STAT 23:get_hits 0
STAT 23:cmd_set 0
STAT 23:delete_hits 0
STAT 23:incr_hits 0
STAT 23:decr_hits 0
STAT 23:cas_hits 0
STAT 23:cas_badval 0
STAT 23:touch_hits 0
STAT 24:chunk_size 17352
STAT 24:chunks_per_page 60
STAT 24:total_pages 1
STAT 24:total_chunks 60
STAT 24:used_chunks 0
STAT 24:free_chunks 60
STAT 24:free_chunks_end 0
STAT 24:get_hits 0
STAT 24:cmd_set 0
STAT 24:delete_hits 0
STAT 24:incr_hits 0
STAT 24:decr_hits 0
STAT 24:cas_hits 0
STAT 24:cas_badval 0
STAT 24:touch_hits 0
STAT 25:chunk_size 21696
STAT 25:chunks_per_page 48
STAT 25:total_pages 1
STAT 25:total_chunks 48
STAT 25:used_chunks 0
STAT 25:free_chunks 48
STAT 25:free_chunks_end 0
STAT 25:get_hits 0
STAT 25:cmd_set 0
STAT 25:delete_hits 0
STAT 25:incr_hits 0
STAT 25:decr_hits 0
STAT 25:cas_hits 0
STAT 25:cas_badval 0
STAT 25:touch_hits 0
STAT 26:chunk_size 27120
STAT 26:chunks_per_page 38
STAT 26:total_pages 1
STAT 26:total_chunks 38
STAT 26:used_chunks 0
STAT 26:free_chunks 38
STAT 26:free_chunks_end 0
STAT 26:get_hits 0
STAT 26:cmd_set 0
STAT 26:delete_hits 0
STAT 26:incr_hits 0
STAT 26:decr_hits 0
STAT 26:cas_hits 0
STAT 26:cas_badval 0
STAT 26:touch_hits 0
STAT 27:chunk_size 33904
STAT 27:chunks_per_page 30
STAT 27:total_pages 1
STAT 27:total_chunks 30
STAT 27:used_chunks 0
STAT 27:free_chunks 30
STAT 27:free_chunks_end 0
STAT 27:get_hits 0
STAT 27:cmd_set 0
STAT 27:delete_hits 0
STAT 27:incr_hits 0
STAT 27:decr_hits 0
STAT 27:cas_hits 0
STAT 27:cas_badval 0
STAT 27:touch_hits 0
STAT 28:chunk_size 42384
STAT 28:chunks_per_page 24
STAT 28:total_pages 1
STAT 28:total_chunks 24
STAT 28:used_chunks 0
STAT 28:free_chunks 24
STAT 28:free_chunks_end 0
STAT 28:get_hits 0
STAT 28:cmd_set 0
STAT 28:delete_hits 0
STAT 28:incr_hits 0
STAT 28:decr_hits 0
STAT 28:cas_hits 0
STAT 28:cas_badval 0
STAT 28:touch_hits 0
STAT 29:chunk_size 52984
STAT 29:chunks_per_page 19
STAT 29:total_pages 1
STAT 29:total_chunks 19
STAT 29:used_chunks 0
STAT 29:free_chunks 19
STAT 29:free_chunks_end 0
STAT 29:get_hits 0
STAT 29:cmd_set 0
STAT 29:delete_hits 0
STAT 29:incr_hits 0
STAT 29:decr_hits 0
STAT 29:cas_hits 0
STAT 29:cas_badval 0
STAT 29:touch_hits 0
STAT 30:chunk_size 66232
STAT 30:chunks_per_page 15
STAT 30:total_pages 1
STAT 30:total_chunks 15
STAT 30:used_chunks 0
STAT 30:free_chunks 15
STAT 30:free_chunks_end 0
STAT 30:get_hits 0
STAT 30:cmd_set 0
STAT 30:delete_hits 0
STAT 30:incr_hits 0
STAT 30:decr_hits 0
STAT 30:cas_hits 0
STAT 30:cas_badval 0
STAT 30:touch_hits 0
STAT 31:chunk_size 82792
STAT 31:chunks_per_page 12
STAT 31:total_pages 1
STAT 31:total_chunks 12
STAT 31:used_chunks 0
STAT 31:free_chunks 12
STAT 31:free_chunks_end 0
STAT 31:get_hits 0
STAT 31:cmd_set 0
STAT 31:delete_hits 0
STAT 31:incr_hits 0
STAT 31:decr_hits 0
STAT 31:cas_hits 0
STAT 31:cas_badval 0
STAT 31:touch_hits 0
STAT 32:chunk_size 103496
STAT 32:chunks_per_page 10
STAT 32:total_pages 1
STAT 32:total_chunks 10
STAT 32:used_chunks 0
STAT 32:free_chunks 10
STAT 32:free_chunks_end 0
STAT 32:get_hits 0
STAT 32:cmd_set 0
STAT 32:delete_hits 0
STAT 32:incr_hits 0
STAT 32:decr_hits 0
STAT 32:cas_hits 0
STAT 32:cas_badval 0
STAT 32:touch_hits 0
STAT 33:chunk_size 129376
STAT 33:chunks_per_page 8
STAT 33:total_pages 1
STAT 33:total_chunks 8
STAT 33:used_chunks 0
STAT 33:free_chunks 8
STAT 33:free_chunks_end 0
STAT 33:get_hits 0
STAT 33:cmd_set 0
STAT 33:delete_hits 0
STAT 33:incr_hits 0
STAT 33:decr_hits 0
STAT 33:cas_hits 0
STAT 33:cas_badval 0
STAT 33:touch_hits 0
STAT 34:chunk_size 161720
STAT 34:chunks_per_page 6
STAT 34:total_pages 1
STAT 34:total_chunks 6
STAT 34:used_chunks 0
STAT 34:free_chunks 6
STAT 34:free_chunks_end 0
STAT 34:get_hits 0
STAT 34:cmd_set 0
STAT 34:delete_hits 0
STAT 34:incr_hits 0
STAT 34:decr_hits 0
STAT 34:cas_hits 0
STAT 34:cas_badval 0
STAT 34:touch_hits 0
STAT 35:chunk_size 202152
STAT 35:chunks_per_page 5
STAT 35:total_pages 1
STAT 35:total_chunks 5
STAT 35:used_chunks 0
STAT 35:free_chunks 5
STAT 35:free_chunks_end 0
STAT 35:get_hits 0
STAT 35:cmd_set 0
STAT 35:delete_hits 0
STAT 35:incr_hits 0
STAT 35:decr_hits 0
STAT 35:cas_hits 0
STAT 35:cas_badval 0
STAT 35:touch_hits 0
STAT 36:chunk_size 252696
STAT 36:chunks_per_page 4
STAT 36:total_pages 1
STAT 36:total_chunks 4
STAT 36:used_chunks 0
STAT 36:free_chunks 4
STAT 36:free_chunks_end 0
STAT 36:get_hits 0
STAT 36:cmd_set 0
STAT 36:delete_hits 0
STAT 36:incr_hits 0
STAT 36:decr_hits 0
STAT 36:cas_hits 0
STAT 36:cas_badval 0
STAT 36:touch_hits 0
STAT 37:chunk_size 315872
STAT 37:chunks_per_page 3
STAT 37:total_pages 1
STAT 37:total_chunks 3
STAT 37:used_chunks 0
STAT 37:free_chunks 3
STAT 37:free_chunks_end 0
STAT 37:get_hits 0
STAT 37:cmd_set 0
STAT 37:delete_hits 0
STAT 37:incr_hits 0
STAT 37:decr_hits 0
STAT 37:cas_hits 0
STAT 37:cas_badval 0
STAT 37:touch_hits 0
STAT 38:chunk_size 394840
STAT 38:chunks_per_page 2
STAT 38:total_pages 1
STAT 38:total_chunks 2
STAT 38:used_chunks 0
STAT 38:free_chunks 2
STAT 38:free_chunks_end 0
STAT 38:get_hits 0
STAT 38:cmd_set 0
STAT 38:delete_hits 0
STAT 38:incr_hits 0
STAT 38:decr_hits 0
STAT 38:cas_hits 0
STAT 38:cas_badval 0
STAT 38:touch_hits 0
STAT 39:chunk_size 524288
STAT 39:chunks_per_page 2
STAT 39:total_pages 1
STAT 39:total_chunks 2
STAT 39:used_chunks 0
STAT 39:free_chunks 2
STAT 39:free_chunks_end 0
STAT 39:get_hits 0
STAT 39:cmd_set 0
STAT 39:delete_hits 0
STAT 39:incr_hits 0
STAT 39:decr_hits 0
STAT 39:cas_hits 0
STAT 39:cas_badval 0
STAT 39:touch_hits 0
STAT active_slabs 39
STAT total_malloced 40894464
END

3. arcus memcached, preallocate X

$ stats slabs
STAT SM:used_num_classes 0
STAT SM:free_num_classes 0
STAT SM:used_min_classid 709
STAT SM:used_01p_classid -1
STAT SM:used_max_classid -1
STAT SM:free_min_classid 709
STAT SM:free_max_classid -1
STAT SM:used_total_space 0
STAT SM:used_01pct_space 0
STAT SM:free_small_space 0
STAT SM:free_bslot_space 0
STAT SM:free_avail_space 0
STAT SM:free_chunk_space 0
STAT SM:free_limit_space 0
STAT SM:space_shortage_level 0
STAT active_slabs 0
STAT memory_limit 67108864
STAT total_malloced 0
END

$ set 01 0 0 2
01
STORED

$ stats slabs
STAT SM:used_num_classes 1
STAT SM:free_num_classes 0
STAT SM:used_min_classid 11
STAT SM:used_01p_classid 11
STAT SM:used_max_classid 11
STAT SM:free_min_classid 709
STAT SM:free_max_classid -1
STAT SM:used_total_space 88
STAT SM:used_01pct_space 88
STAT SM:free_small_space 0
STAT SM:free_bslot_space 262024
STAT SM:free_avail_space 262024
STAT SM:free_chunk_space 0
STAT SM:free_limit_space 0
STAT SM:space_shortage_level 0
STAT 0:chunk_size 262144
STAT 0:chunks_per_page 4
STAT 0:reserved_pages 0
STAT 0:total_pages 1
STAT 0:total_chunks 4
STAT 0:used_chunks 1
STAT 0:free_chunks 0
STAT 0:free_chunks_end 3
STAT 0:mem_requested 262144
STAT active_slabs 1
STAT memory_limit 67108864
STAT total_malloced 1048576
END

4. arcus memcached, preallocate O

stats slabs
STAT SM:used_num_classes 0
STAT SM:free_num_classes 0
STAT SM:used_min_classid 709
STAT SM:used_01p_classid -1
STAT SM:used_max_classid -1
STAT SM:free_min_classid 709
STAT SM:free_max_classid -1
STAT SM:used_total_space 0
STAT SM:used_01pct_space 0
STAT SM:free_small_space 0
STAT SM:free_bslot_space 0
STAT SM:free_avail_space 0
STAT SM:free_chunk_space 0
STAT SM:free_limit_space 0
STAT SM:space_shortage_level 0
STAT 1:chunk_size 112
STAT 1:chunks_per_page 9362
STAT 1:reserved_pages 4
STAT 1:total_pages 1
STAT 1:total_chunks 9362
STAT 1:used_chunks 0
STAT 1:free_chunks 0
STAT 1:free_chunks_end 9362
STAT 1:mem_requested 0
STAT 2:chunk_size 144
STAT 2:chunks_per_page 7281
STAT 2:reserved_pages 4
STAT 2:total_pages 1
STAT 2:total_chunks 7281
STAT 2:used_chunks 0
STAT 2:free_chunks 0
STAT 2:free_chunks_end 7281
STAT 2:mem_requested 0
STAT 3:chunk_size 184
STAT 3:chunks_per_page 5698
STAT 3:reserved_pages 4
STAT 3:total_pages 1
STAT 3:total_chunks 5698
STAT 3:used_chunks 0
STAT 3:free_chunks 0
STAT 3:free_chunks_end 5698
STAT 3:mem_requested 0
STAT 4:chunk_size 232
STAT 4:chunks_per_page 4519
STAT 4:reserved_pages 4
STAT 4:total_pages 1
STAT 4:total_chunks 4519
STAT 4:used_chunks 0
STAT 4:free_chunks 0
STAT 4:free_chunks_end 4519
STAT 4:mem_requested 0
STAT 5:chunk_size 296
STAT 5:chunks_per_page 3542
STAT 5:reserved_pages 4
STAT 5:total_pages 1
STAT 5:total_chunks 3542
STAT 5:used_chunks 0
STAT 5:free_chunks 0
STAT 5:free_chunks_end 3542
STAT 5:mem_requested 0
STAT 6:chunk_size 376
STAT 6:chunks_per_page 2788
STAT 6:reserved_pages 4
STAT 6:total_pages 1
STAT 6:total_chunks 2788
STAT 6:used_chunks 0
STAT 6:free_chunks 0
STAT 6:free_chunks_end 2788
STAT 6:mem_requested 0
STAT 7:chunk_size 472
STAT 7:chunks_per_page 2221
STAT 7:reserved_pages 4
STAT 7:total_pages 1
STAT 7:total_chunks 2221
STAT 7:used_chunks 0
STAT 7:free_chunks 0
STAT 7:free_chunks_end 2221
STAT 7:mem_requested 0
STAT 8:chunk_size 592
STAT 8:chunks_per_page 1771
STAT 8:reserved_pages 4
STAT 8:total_pages 1
STAT 8:total_chunks 1771
STAT 8:used_chunks 0
STAT 8:free_chunks 0
STAT 8:free_chunks_end 1771
STAT 8:mem_requested 0
STAT 9:chunk_size 744
STAT 9:chunks_per_page 1409
STAT 9:reserved_pages 4
STAT 9:total_pages 1
STAT 9:total_chunks 1409
STAT 9:used_chunks 0
STAT 9:free_chunks 0
STAT 9:free_chunks_end 1409
STAT 9:mem_requested 0
STAT 10:chunk_size 936
STAT 10:chunks_per_page 1120
STAT 10:reserved_pages 4
STAT 10:total_pages 1
STAT 10:total_chunks 1120
STAT 10:used_chunks 0
STAT 10:free_chunks 0
STAT 10:free_chunks_end 1120
STAT 10:mem_requested 0
STAT 11:chunk_size 1176
STAT 11:chunks_per_page 891
STAT 11:reserved_pages 4
STAT 11:total_pages 1
STAT 11:total_chunks 891
STAT 11:used_chunks 0
STAT 11:free_chunks 0
STAT 11:free_chunks_end 891
STAT 11:mem_requested 0
STAT 12:chunk_size 1472
STAT 12:chunks_per_page 712
STAT 12:reserved_pages 4
STAT 12:total_pages 1
STAT 12:total_chunks 712
STAT 12:used_chunks 0
STAT 12:free_chunks 0
STAT 12:free_chunks_end 712
STAT 12:mem_requested 0
STAT 13:chunk_size 1840
STAT 13:chunks_per_page 569
STAT 13:reserved_pages 4
STAT 13:total_pages 1
STAT 13:total_chunks 569
STAT 13:used_chunks 0
STAT 13:free_chunks 0
STAT 13:free_chunks_end 569
STAT 13:mem_requested 0
STAT 14:chunk_size 2304
STAT 14:chunks_per_page 455
STAT 14:reserved_pages 4
STAT 14:total_pages 1
STAT 14:total_chunks 455
STAT 14:used_chunks 0
STAT 14:free_chunks 0
STAT 14:free_chunks_end 455
STAT 14:mem_requested 0
STAT 15:chunk_size 2880
STAT 15:chunks_per_page 364
STAT 15:reserved_pages 4
STAT 15:total_pages 1
STAT 15:total_chunks 364
STAT 15:used_chunks 0
STAT 15:free_chunks 0
STAT 15:free_chunks_end 364
STAT 15:mem_requested 0
STAT 16:chunk_size 3600
STAT 16:chunks_per_page 291
STAT 16:reserved_pages 4
STAT 16:total_pages 1
STAT 16:total_chunks 291
STAT 16:used_chunks 0
STAT 16:free_chunks 0
STAT 16:free_chunks_end 291
STAT 16:mem_requested 0
STAT 17:chunk_size 4504
STAT 17:chunks_per_page 232
STAT 17:reserved_pages 4
STAT 17:total_pages 1
STAT 17:total_chunks 232
STAT 17:used_chunks 0
STAT 17:free_chunks 0
STAT 17:free_chunks_end 232
STAT 17:mem_requested 0
STAT 18:chunk_size 5632
STAT 18:chunks_per_page 186
STAT 18:reserved_pages 4
STAT 18:total_pages 1
STAT 18:total_chunks 186
STAT 18:used_chunks 0
STAT 18:free_chunks 0
STAT 18:free_chunks_end 186
STAT 18:mem_requested 0
STAT 19:chunk_size 7040
STAT 19:chunks_per_page 148
STAT 19:reserved_pages 4
STAT 19:total_pages 1
STAT 19:total_chunks 148
STAT 19:used_chunks 0
STAT 19:free_chunks 0
STAT 19:free_chunks_end 148
STAT 19:mem_requested 0
STAT 20:chunk_size 8800
STAT 20:chunks_per_page 119
STAT 20:reserved_pages 4
STAT 20:total_pages 1
STAT 20:total_chunks 119
STAT 20:used_chunks 0
STAT 20:free_chunks 0
STAT 20:free_chunks_end 119
STAT 20:mem_requested 0
STAT 21:chunk_size 11000
STAT 21:chunks_per_page 95
STAT 21:reserved_pages 4
STAT 21:total_pages 1
STAT 21:total_chunks 95
STAT 21:used_chunks 0
STAT 21:free_chunks 0
STAT 21:free_chunks_end 95
STAT 21:mem_requested 0
STAT 22:chunk_size 13752
STAT 22:chunks_per_page 76
STAT 22:reserved_pages 4
STAT 22:total_pages 1
STAT 22:total_chunks 76
STAT 22:used_chunks 0
STAT 22:free_chunks 0
STAT 22:free_chunks_end 76
STAT 22:mem_requested 0
STAT 23:chunk_size 17192
STAT 23:chunks_per_page 60
STAT 23:reserved_pages 4
STAT 23:total_pages 1
STAT 23:total_chunks 60
STAT 23:used_chunks 0
STAT 23:free_chunks 0
STAT 23:free_chunks_end 60
STAT 23:mem_requested 0
STAT 24:chunk_size 21496
STAT 24:chunks_per_page 48
STAT 24:reserved_pages 4
STAT 24:total_pages 1
STAT 24:total_chunks 48
STAT 24:used_chunks 0
STAT 24:free_chunks 0
STAT 24:free_chunks_end 48
STAT 24:mem_requested 0
STAT 25:chunk_size 26872
STAT 25:chunks_per_page 39
STAT 25:reserved_pages 4
STAT 25:total_pages 1
STAT 25:total_chunks 39
STAT 25:used_chunks 0
STAT 25:free_chunks 0
STAT 25:free_chunks_end 39
STAT 25:mem_requested 0
STAT 26:chunk_size 33592
STAT 26:chunks_per_page 31
STAT 26:reserved_pages 4
STAT 26:total_pages 1
STAT 26:total_chunks 31
STAT 26:used_chunks 0
STAT 26:free_chunks 0
STAT 26:free_chunks_end 31
STAT 26:mem_requested 0
STAT 27:chunk_size 41992
STAT 27:chunks_per_page 24
STAT 27:reserved_pages 4
STAT 27:total_pages 1
STAT 27:total_chunks 24
STAT 27:used_chunks 0
STAT 27:free_chunks 0
STAT 27:free_chunks_end 24
STAT 27:mem_requested 0
STAT 28:chunk_size 52496
STAT 28:chunks_per_page 19
STAT 28:reserved_pages 4
STAT 28:total_pages 1
STAT 28:total_chunks 19
STAT 28:used_chunks 0
STAT 28:free_chunks 0
STAT 28:free_chunks_end 19
STAT 28:mem_requested 0
STAT 29:chunk_size 65624
STAT 29:chunks_per_page 15
STAT 29:reserved_pages 4
STAT 29:total_pages 1
STAT 29:total_chunks 15
STAT 29:used_chunks 0
STAT 29:free_chunks 0
STAT 29:free_chunks_end 15
STAT 29:mem_requested 0
STAT 30:chunk_size 82032
STAT 30:chunks_per_page 12
STAT 30:reserved_pages 4
STAT 30:total_pages 1
STAT 30:total_chunks 12
STAT 30:used_chunks 0
STAT 30:free_chunks 0
STAT 30:free_chunks_end 12
STAT 30:mem_requested 0
STAT 31:chunk_size 102544
STAT 31:chunks_per_page 10
STAT 31:reserved_pages 4
STAT 31:total_pages 1
STAT 31:total_chunks 10
STAT 31:used_chunks 0
STAT 31:free_chunks 0
STAT 31:free_chunks_end 10
STAT 31:mem_requested 0
STAT 32:chunk_size 128184
STAT 32:chunks_per_page 8
STAT 32:reserved_pages 4
STAT 32:total_pages 1
STAT 32:total_chunks 8
STAT 32:used_chunks 0
STAT 32:free_chunks 0
STAT 32:free_chunks_end 8
STAT 32:mem_requested 0
STAT 33:chunk_size 160232
STAT 33:chunks_per_page 6
STAT 33:reserved_pages 4
STAT 33:total_pages 1
STAT 33:total_chunks 6
STAT 33:used_chunks 0
STAT 33:free_chunks 0
STAT 33:free_chunks_end 6
STAT 33:mem_requested 0
STAT 34:chunk_size 200296
STAT 34:chunks_per_page 5
STAT 34:reserved_pages 4
STAT 34:total_pages 1
STAT 34:total_chunks 5
STAT 34:used_chunks 0
STAT 34:free_chunks 0
STAT 34:free_chunks_end 5
STAT 34:mem_requested 0
STAT 35:chunk_size 250376
STAT 35:chunks_per_page 4
STAT 35:reserved_pages 4
STAT 35:total_pages 1
STAT 35:total_chunks 4
STAT 35:used_chunks 0
STAT 35:free_chunks 0
STAT 35:free_chunks_end 4
STAT 35:mem_requested 0
STAT 36:chunk_size 312976
STAT 36:chunks_per_page 3
STAT 36:reserved_pages 4
STAT 36:total_pages 1
STAT 36:total_chunks 3
STAT 36:used_chunks 0
STAT 36:free_chunks 0
STAT 36:free_chunks_end 3
STAT 36:mem_requested 0
STAT 37:chunk_size 391224
STAT 37:chunks_per_page 2
STAT 37:reserved_pages 4
STAT 37:total_pages 1
STAT 37:total_chunks 2
STAT 37:used_chunks 0
STAT 37:free_chunks 0
STAT 37:free_chunks_end 2
STAT 37:mem_requested 0
STAT 38:chunk_size 489032
STAT 38:chunks_per_page 2
STAT 38:reserved_pages 4
STAT 38:total_pages 1
STAT 38:total_chunks 2
STAT 38:used_chunks 0
STAT 38:free_chunks 0
STAT 38:free_chunks_end 2
STAT 38:mem_requested 0
STAT 39:chunk_size 611296
STAT 39:chunks_per_page 1
STAT 39:reserved_pages 6
STAT 39:total_pages 1
STAT 39:total_chunks 1
STAT 39:used_chunks 0
STAT 39:free_chunks 0
STAT 39:free_chunks_end 1
STAT 39:mem_requested 0
STAT 40:chunk_size 764120
STAT 40:chunks_per_page 1
STAT 40:reserved_pages 6
STAT 40:total_pages 1
STAT 40:total_chunks 1
STAT 40:used_chunks 0
STAT 40:free_chunks 0
STAT 40:free_chunks_end 1
STAT 40:mem_requested 0
STAT 41:chunk_size 1048576
STAT 41:chunks_per_page 1
STAT 41:reserved_pages 6
STAT 41:total_pages 1
STAT 41:total_chunks 1
STAT 41:used_chunks 0
STAT 41:free_chunks 0
STAT 41:free_chunks_end 1
STAT 41:mem_requested 0
STAT active_slabs 41
STAT memory_limit 67108864
STAT total_malloced 41308144
END

[namsic]

@6unYoung
slab preallocate가 어떤 목적으로 사용되고, 어떤 동작을 하는지 (개념 수준에서) 정리해줄 수 있나요?

[6unYoung]

• preallocate 사용 시 서버를 시작할 때 메모리를 통째로 미리 할당해 두므로, 데이터를 저장할 때 메모리 부족으로 새로운 slab를 할당하는 오버헤드가 없습니다. 또한, OS 레벨에서 메모리 단편화로 생기는 메모리 부족 현상이 없습니다.
• 'L' 옵션을 주면 slabs_init()에서 slabs_preallocate()를 호출합니다. 모든 slab class별로 slab 할당을 시도합니다. 각 slab는 class에 설정된 크기대로 잘게 쪼개지고 이때 조각 하나가 chunk가 됩니다. 남은 메모리 공간이 부족하면 할당을 중단합니다. class id 0은 small memory 공간으로 사용합니다.

[jhpark816]

@6unYoung
OSS memcached도 아래 style로 코딩되어 있나요?

#ifndef DONT_PREALLOC_SLABS
. . . 
#endif

참고로, 아래 형태의 코드가 더 낫다고 봅니다.

#ifdef DO_PREALLOC_SLABS
. . . 
#endif

[6unYoung]

@6unYoung OSS memcached도 아래 style로 코딩되어 있나요?

#ifndef DONT_PREALLOC_SLABS
. . .
#endif
참고로, 아래 형태의 코드가 더 낫다고 봅니다.

#ifdef DO_PREALLOC_SLABS
. . .
#endif

OSS memcached에서는 전처리 태그를 사용하지 않습니다.

// 대략적인 흐름만 나타낸 것입니다.
void slabs_init(const bool prealloc, ...) {
    if (prealloc) {
        do_slab_prealloc = true ;
    }
    if (do_slab_prealloc) {
        slabs_preallocate();
    }
}

[6unYoung]

결론

현재 '-L' 옵션을 줘도 preallocate가 수행되지 않습니다.
원인은 다음과 같습니다.

1. slabs_preallocate() 호출 조건이 옵션과 무관하게 환경변수로만 비교하고 있습니다.
2. slabs_preallocate() 호출부를 감싼 DONT_PREALLOC_SLABS 태그가 활성화되어 있습니다.

다음과 같이 수정하면 '-L' 옵션에 의해 preallocate가 작동합니다.
문제 없으면 이대로 수정 진행하겠습니다.

// slabs.c
- #define DONT_PREALLOC_SLABS
- #ifndef DONT_PREALLOC_SLABS
-    {
-        char *pre_alloc = getenv("T_MEMD_SLABS_ALLOC");
-
-        if (pre_alloc == NULL || atoi(pre_alloc) != 0) {
-            slabs_preallocate(power_largest);
-        }
-    }
+ #define DO_PREALLOC_SLABS
+ #ifdef DO_PREALLOC_SLABS
+   {
+        if (prealloc) {
+            slabs_preallocate(slabsp->power_largest);
+        }
+    }
#endif 

// slabs.c
#ifndef DONT_PREALLOC_SLABS
static void slabs_preallocate(const unsigned int maxslabs)
{
    int i;
    unsigned int prealloc = 0;

    for (i = POWER_SMALLEST; i <= POWER_LARGEST; i++) {
        if (++prealloc > maxslabs)
            return;
-          do_slabs_newslab(i);
+       if (do_slabs_newslab(i) == 0) {
+           // error handling;
+           return;
+       }
    }

    /* slab class 0 is used for collection items and small-size kv items */
    do_slabs_newslab(0);
}
#endif

[jhpark816]

수정 방안에 대해 왜 수정해야 하는 지를 설명해 주세요.

• preallocate 변수는 사용되지도 않을 건데, 있을 필요가 있나요?
• do_slabs_newslab() 수행이 실패하더라도 exit() 호출하면 안 됩니다. OSS memcached에서는 어떻게 처리하는 지 ?
  • 실패하면, prealloc 하지 않는 상태로 동작하면 됩니다.
  • 실패시에 구동을 중지하고 싶다면, exit() 직접하는 것이 아니라 수행 상태를 리턴하고 memcached.c 파일에서 최종적으로 exit() 호출하게 해야 합니다.
• DO_PREALLOC_SLABS 플래그로 변경은 보류합시다.
  플래그를 두는 것이 나은 지? 아니면 OSS memcached 구현이 나은 지 ? 의견 주세요.

[6unYoung]

T_MEMD_SLABS_ALLOC 환경변수에 대해서 조사해 본 결과 OSS memcached에서는 찾을 수 없었고, arcus 내부에서도 사용처가 따로 없습니다. arcus 내부에서 '-L' 옵션 대신 환경 변수로 test를 하기 위한 임시 변수로 추측됩니다.
동일하게 T_MEMD_INITIAL_MALLOC 환경변수도 slabs_init() 내부에서 test suite를 위해 사용되고 있습니다.

[6unYoung]

수정 방안에 대해 왜 수정해야 하는 지를 설명해 주세요.

• preallocate 변수는 사용되지도 않을 건데, 있을 필요가 있나요?

본래 'L' 옵션을 준 경우에 preallocate을 받아 바로 slabs_init()에 넘기기 위한 용도인데 현재는 사용되지 않는 것으로 보여 삭제하겠습니다.

• do_slabs_newslab() 수행 시에 exit() 호출하면 안 됩니다. OSS memcached에서는 어떻게 처리하는 지 ?

해당 코드는 OSS memcached와 동일합니다.

• 실패하면, prealloc 하지 않는 상태로 동작하면 됩니다.

do_slabs_newslab()를 반복 호출하다가 어느 시점에서 실패했을 때, prealloc 하지 않는 상태로 바꾸는 것은 이미 할당한 메모리와 prealloc config 설정을 롤백하는 오버헤드가 크고 자연스러운 흐름도 아니라 구동을 중지하는 게 맞는 것 같습니다.

• 실패시에 구동을 중지하고 싶다면, exit() 직접하는 것이 아니라 상태를 리턴하고 memcached.c 파일에서 최종적으로 exit() 호출하게 해야 합니다.

event loop 전에 main()->init_engine()->initialize()->slabs_init() 스택으로 호출 중입니다. 이때 반환형이 ENGINE_ERROR_CODE으로 init_engine()에서 error를 받아 처리하고 있습니다. 그런데 OSS memcached에서는 1MB씩 preallocate하는 과정에서 실패하는 경우를 ENOMEM를 구분하려고 이 경우에는 slabs_preallocate() 내부에서 에러처리한 것입니다.
말씀하신 방식으로 하게 된다면 slab_preallocate()에서는 에러 메시지만 출력하고 ENGINE_FAILED로 반환해 일반적인 실패 흐름이 되게 해야 할 것 같습니다.

• DO_PREALLOC_SLABS 플래그로 변경은 보류합시다.
  플래그를 두는 것이 나은 지? 아니면 OSS memcached 구현이 나은 지 ? 의견 주세요.

정상 작동하게 수정하면 플래그는 불필요하므로 삭제하는 게 나을 것 같습니다.