Newshot Memory management in 40 major C projects, a survey
A survey of 40 major C projects tracked by The Consensus shows only 8 projects allocate solely with malloc/free, only 2 projects allocate without ever using malloc/free, 15 projects make use of regions at some point, 7 projects primarily use regions to allocate, 5 primarily use garbage collection to allocate, and 3 primarily use reference counting to allocate.
You are getting early access to this article as a subscriber. Your support makes articles like this possible. Thank you.
C is well known for requiring manual memory management with malloc and free. And while many major projects do actually manage memory like this, many projects also implement higher-level memory management primitives such as regions (where objects allocated within a region need not be manually freed, freeing a region frees all objects allocated within the region at once), reference counting (manually incrementing and decrementing outstanding references, freeing an object only when its references go to zero), scope-based cleanup using compiler extensions, and a number of major C projects even actually implement garbage collection (fully automatic memory management).
Programmers sometimes assume that projects famous for diligence lean away from purely manual memory management. But SQLite, famous for correctness, allocates 100% manually while cURL and Redis allocate mostly manually (94% and 79%), reference counting only some of their objects. Another surprising finding is that Linux and QEMU in the last few years have been starting to use scope cleanup rules (i.e. compiler extensions) to automatically clean up memory, such as Linux's __free(kfree) and QEMU's g_autofree.
The Consensus tracks and indexes code for over 400 major software infrastructure projects. This article surveys 40 of the C projects within this group, excluding test code, third-party code, and examples.
Majority allocation strategy
To discover majority allocation we count allocation call sites (which is certainly a limited metric, not accounting for sub-region or size of allocation, etc.). seL4 is an exception to much of this article since it has no heap at all.
8 projects solely allocate with malloc, implementing no other general higher-level strategy for allocating. These projects are: brotli, FreeRTOS, HAProxy, libdeflate, lz4, SQLite, xz-utils, and zlib.
16 projects primarily, but not exclusively, allocate with malloc.
| Project | Manual % | Region % | Garbage Collector % | Reference Counting % |
|---|---|---|---|---|
| PgBouncer | 98 | 0 | 0 | 2 |
| tmux | 98 | 0 | 0 | 2 |
| git | 96 | 3 | 0 | 0 |
| curl | 94 | 0 | 0 | 6 |
| dqlite | 92 | 0 | 0 | 8 |
| QEMU | 86 | 0 | 0 | 14 |
| Vim | 85 | 0 | 0 | 15 |
| Redis | 79 | 0 | 0 | 21 |
| Neovim | 74 | 10 | 0 | 15 |
| universal-ctags | 72 | 0 | 0 | 28 |
| Linux | 71 | 29 | 0 | 0 |
| zstd | 70 | 30 | 0 | 0 |
| tree-sitter | 62 | 0 | 0 | 38 |
| Erlang | 57 | 0 | 39 | 4 |
| PostGIS | 54 | 46 | 0 | 1 |
| Xen | 51 | 49 | 0 | 0 |
7 projects primarily, if not all exclusively, allocate with a region system. With region based allocators, objects are allocated within a region. Freeing the region frees all objects within the region. Sometiems regions can be nested. And sometimes you might still manually free an object allocated within a region.
| Project | Region % | Manual % | Garbage Collector % |
|---|---|---|---|
| pgvector | 100 | 0 | 0 |
| TimescaleDB | 100 | 0 | 0 |
| Citus | 99 | 1 | 0 |
| httpd | 98 | 2 | 0 |
| nginx | 92 | 8 | 0 |
| Postgres | 78 | 22 | 0 |
| llama.cpp (ggml) | 65 | 35 | 0 |
And 5 projects primarily allocate with a garbage collector.
| Project | Garbage Collector % | Manual % | Region % |
|---|---|---|---|
| Ruby | 82 | 18 | 0 |
| Julia | 80 | 20 | 0 |
| Emacs | 77 | 23 | 0 |
| OCaml | 62 | 38 | 0 |
| Lua | 60 | 40 | 0 |
3 projects primarily allocate reference-counted objects.
| Project | Reference Counting % | Manual % | Region % |
|---|---|---|---|
| CPython | 85 | 13 | 2 |
| PHP | 62 | 36 | 2 |
| Perl | 62 | 38 | 0 |
All projects
Across all surveyed projects, with links to their various allocation strategies by example.
| Project | free/malloc | Region | Garbage Collector | Reference Counts | Scope Cleanup |
|---|---|---|---|---|---|
| sqlite | sqlite3_mprintf , sqlite3_free | none | none | none | none |
| haproxy | strdup , free | none | none | none | none |
| pgbouncer | malloc , free | none | none | strpool_decref | none |
| redis | zmalloc , zfree | none | none | decrRefCount | none |
| git | xstrdup , free | mem_pool_alloc , mem_pool_init | none | blame_origin_decref | none |
| tmux | xstrdup , free | none | none | server_client_unref | none |
| vim | vim_strsave , vim_free | none | garbage_collect | dict_unref | none |
| neovim | xstrdup , xfree | arena_memdupz , arena_mem_free | none | tv_dict_unref | none |
| tree-sitter | ts_malloc , ts_free | none | none | ts_subtree_release | none |
| universal-ctags | eStrdup , eFree | none | none | es_object_unref | none |
| zstandard | malloc , free | ZSTD_cwksp_reserve_aligned64 , ZSTD_cwksp_init | none | none | none |
| zlib | ZALLOC , ZFREE | none | none | none | none |
| xz-utils | lzma_alloc , lzma_free | none | none | none | none |
| lz4 | ALLOC , FREEMEM | none | none | none | none |
| brotli | BROTLI_ALLOC , BROTLI_FREE | none | none | none | none |
| libdeflate | libdeflate_aligned_malloc , libdeflate_aligned_free | none | none | none | none |
| freertos | pvPortMalloc , vPortFree | none | none | none | none |
| dqlite | raft_malloc , raft_free | none | none | logRelease | none |
| curl | curlx_calloc , curlx_free | none | none | Curl_creds_unlink | none |
| xen | xzalloc , xfree | GCSPRINTF , GC_INIT | none | none | none |
| qemu | g_new0 , g_free | none | none | object_unref | g_autofree |
| linux | kzalloc_obj , kfree | devm_kzalloc | none | kref_put | __free |
| postgresql | pg_strdup , pg_free | palloc , MemoryContextReset | none | none | none |
| pgvector | none | palloc , MemoryContextDelete | none | none | none |
| citus | strdup , free | palloc0 , MemoryContextReset | none | none | none |
| timescaledb | none | palloc , AllocSetContextCreate | none | none | none |
| postgis | lwalloc , lwfree | palloc , AllocSetContextCreate | none | shared_gserialized_unref | none |
| nginx | ngx_alloc , ngx_free | ngx_create_pool , ngx_destroy_pool | none | none | none |
| apache-http-server | apr_bucket_alloc , apr_bucket_free | apr_pool_create , apr_pool_clear | none | none | none |
| llama-cpp (ggml C) | calloc , free | ggml_new_tensor_1d , ggml_free | none | none | none |
Noticed a mistake? Have a question or comment? Write to the editor.