Pushing mmdbshrink further
Published on .
IPinfo’s mmdbshrink is a neat tool for shrinking MaxMind DB (MMDB) files by finding and deduplicating identical search tree subtrees. It reduces file sizes by 30-80% without touching lookup accuracy or requiring any decompression at runtime.
It piqued my interest and I realized that the bit-width of the record pointers might be wasting bytes since the spec allows 24-bit, 28-bit and 32-bit but ipinfo’s ipinfo_lite.mmdb was using 32-bit pointers for just 2,396,775 nodes.
gpt-6-luna authored a nice 32 line diff to implement repacking to the smallest possible bit-width. See https://github.com/ipinfo/mmdbctl/pull/60/changes and it reduced the filesize by 18%
The Results on ipinfo_lite
input: ipinfo_lite.mmdb
output: ipinfo_lite.shrunk.mmdb
size: 26.34 MB -> 21.55 MB (saved 4.79 MB, -18.20%)
nodes: 2,396,775 -> 2,396,775 (unchanged)
tree: 19.17 MB -> 14.38 MB
data: 7.17 MB -> 7.17 MB (unchanged)
Code language: Python (python)By safely dialing down the record_size from 32 to 24, the search tree footprint dropped from 19.17 MB to 14.38 MB, achieving an additional 18.20% reduction—all while keeping lookups fully validated and lightning fast (645k+ QPS).