@matheusbsbs / matheus-bagliano / commits / 6a364ced49

add a hashtable implementation that supports O(1) removal

The existing hashtable implementation (in hash.[ch]) uses open addressing (i.e. resolve hash collisions by distributing entries across the table). Thus, removal is difficult to implement with less than O(n) complexity. Resolving collisions of entries with identical hashes (e.g. via chaining) is left to the client code. Add a hashtable implementation that supports O(1) removal and is slightly easier to use due to builtin entry chaining. Supports all basic operations init, free, get, add, remove and iteration. Also includes ready-to-use hash functions based on the public domain FNV-1 algorithm (http://www.isthe.com/chongo/tech/comp/fnv). The per-entry data structure (hashmap_entry) is piggybacked in front of the client's data structure to save memory. See test-hashmap.c for usage examples. The hashtable is resized by a factor of four when 80% full. With these settings, average memory consumption is about 2/3 of hash.[ch], and insertion is about twice as fast due to less frequent resizing. Lookups are also slightly faster, because entries are strictly confined to their bucket (i.e. no data of other buckets needs to be traversed). Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Karsten Blees committed Nov 14, 2013 at 20:17 UTC 6a364ced497e407ab3ffb2554d4ef2c78f801832
6 files changed +1117
Documentation/technical/api-hashmap.txt new
+235
@@ -0,0 +1,235 @@
1 +hashmap API
2 +===========
3 +
4 +The hashmap API is a generic implementation of hash-based key-value mappings.
5 +
6 +Data Structures
7 +---------------
8 +
9 +`struct hashmap`::
10 +
11 + The hash table structure.
12 ++
13 +The `size` member keeps track of the total number of entries. The `cmpfn`
14 +member is a function used to compare two entries for equality. The `table` and
15 +`tablesize` members store the hash table and its size, respectively.
16 +
17 +`struct hashmap_entry`::
18 +
19 + An opaque structure representing an entry in the hash table, which must
20 + be used as first member of user data structures. Ideally it should be
21 + followed by an int-sized member to prevent unused memory on 64-bit
22 + systems due to alignment.
23 ++
24 +The `hash` member is the entry's hash code and the `next` member points to the
25 +next entry in case of collisions (i.e. if multiple entries map to the same
26 +bucket).
27 +
28 +`struct hashmap_iter`::
29 +
30 + An iterator structure, to be used with hashmap_iter_* functions.
31 +
32 +Types
33 +-----
34 +
35 +`int (*hashmap_cmp_fn)(const void *entry, const void *entry_or_key, const void *keydata)`::
36 +
37 + User-supplied function to test two hashmap entries for equality. Shall
38 + return 0 if the entries are equal.
39 ++
40 +This function is always called with non-NULL `entry` / `entry_or_key`
41 +parameters that have the same hash code. When looking up an entry, the `key`
42 +and `keydata` parameters to hashmap_get and hashmap_remove are always passed
43 +as second and third argument, respectively. Otherwise, `keydata` is NULL.
44 +
45 +Functions
46 +---------
47 +
48 +`unsigned int strhash(const char *buf)`::
49 +`unsigned int strihash(const char *buf)`::
50 +`unsigned int memhash(const void *buf, size_t len)`::
51 +`unsigned int memihash(const void *buf, size_t len)`::
52 +
53 + Ready-to-use hash functions for strings, using the FNV-1 algorithm (see
54 + http://www.isthe.com/chongo/tech/comp/fnv).
55 ++
56 +`strhash` and `strihash` take 0-terminated strings, while `memhash` and
57 +`memihash` operate on arbitrary-length memory.
58 ++
59 +`strihash` and `memihash` are case insensitive versions.
60 +
61 +`void hashmap_init(struct hashmap *map, hashmap_cmp_fn equals_function, size_t initial_size)`::
62 +
63 + Initializes a hashmap structure.
64 ++
65 +`map` is the hashmap to initialize.
66 ++
67 +The `equals_function` can be specified to compare two entries for equality.
68 +If NULL, entries are considered equal if their hash codes are equal.
69 ++
70 +If the total number of entries is known in advance, the `initial_size`
71 +parameter may be used to preallocate a sufficiently large table and thus
72 +prevent expensive resizing. If 0, the table is dynamically resized.
73 +
74 +`void hashmap_free(struct hashmap *map, int free_entries)`::
75 +
76 + Frees a hashmap structure and allocated memory.
77 ++
78 +`map` is the hashmap to free.
79 ++
80 +If `free_entries` is true, each hashmap_entry in the map is freed as well
81 +(using stdlib's free()).
82 +
83 +`void hashmap_entry_init(void *entry, int hash)`::
84 +
85 + Initializes a hashmap_entry structure.
86 ++
87 +`entry` points to the entry to initialize.
88 ++
89 +`hash` is the hash code of the entry.
90 +
91 +`void *hashmap_get(const struct hashmap *map, const void *key, const void *keydata)`::
92 +
93 + Returns the hashmap entry for the specified key, or NULL if not found.
94 ++
95 +`map` is the hashmap structure.
96 ++
97 +`key` is a hashmap_entry structure (or user data structure that starts with
98 +hashmap_entry) that has at least been initialized with the proper hash code
99 +(via `hashmap_entry_init`).
100 ++
101 +If an entry with matching hash code is found, `key` and `keydata` are passed
102 +to `hashmap_cmp_fn` to decide whether the entry matches the key.
103 +
104 +`void *hashmap_get_next(const struct hashmap *map, const void *entry)`::
105 +
106 + Returns the next equal hashmap entry, or NULL if not found. This can be
107 + used to iterate over duplicate entries (see `hashmap_add`).
108 ++
109 +`map` is the hashmap structure.
110 ++
111 +`entry` is the hashmap_entry to start the search from, obtained via a previous
112 +call to `hashmap_get` or `hashmap_get_next`.
113 +
114 +`void hashmap_add(struct hashmap *map, void *entry)`::
115 +
116 + Adds a hashmap entry. This allows to add duplicate entries (i.e.
117 + separate values with the same key according to hashmap_cmp_fn).
118 ++
119 +`map` is the hashmap structure.
120 ++
121 +`entry` is the entry to add.
122 +
123 +`void *hashmap_put(struct hashmap *map, void *entry)`::
124 +
125 + Adds or replaces a hashmap entry. If the hashmap contains duplicate
126 + entries equal to the specified entry, only one of them will be replaced.
127 ++
128 +`map` is the hashmap structure.
129 ++
130 +`entry` is the entry to add or replace.
131 ++
132 +Returns the replaced entry, or NULL if not found (i.e. the entry was added).
133 +
134 +`void *hashmap_remove(struct hashmap *map, const void *key, const void *keydata)`::
135 +
136 + Removes a hashmap entry matching the specified key. If the hashmap
137 + contains duplicate entries equal to the specified key, only one of
138 + them will be removed.
139 ++
140 +`map` is the hashmap structure.
141 ++
142 +`key` is a hashmap_entry structure (or user data structure that starts with
143 +hashmap_entry) that has at least been initialized with the proper hash code
144 +(via `hashmap_entry_init`).
145 ++
146 +If an entry with matching hash code is found, `key` and `keydata` are
147 +passed to `hashmap_cmp_fn` to decide whether the entry matches the key.
148 ++
149 +Returns the removed entry, or NULL if not found.
150 +
151 +`void hashmap_iter_init(struct hashmap *map, struct hashmap_iter *iter)`::
152 +`void *hashmap_iter_next(struct hashmap_iter *iter)`::
153 +`void *hashmap_iter_first(struct hashmap *map, struct hashmap_iter *iter)`::
154 +
155 + Used to iterate over all entries of a hashmap.
156 ++
157 +`hashmap_iter_init` initializes a `hashmap_iter` structure.
158 ++
159 +`hashmap_iter_next` returns the next hashmap_entry, or NULL if there are no
160 +more entries.
161 ++
162 +`hashmap_iter_first` is a combination of both (i.e. initializes the iterator
163 +and returns the first entry, if any).
164 +
165 +Usage example
166 +-------------
167 +
168 +Here's a simple usage example that maps long keys to double values.
169 +[source,c]
170 +------------
171 +struct hashmap map;
172 +
173 +struct long2double {
174 + struct hashmap_entry ent; /* must be the first member! */
175 + long key;
176 + double value;
177 +};
178 +
179 +static int long2double_cmp(const struct long2double *e1, const struct long2double *e2, const void *unused)
180 +{
181 + return !(e1->key == e2->key);
182 +}
183 +
184 +void long2double_init(void)
185 +{
186 + hashmap_init(&map, (hashmap_cmp_fn) long2double_cmp, 0);
187 +}
188 +
189 +void long2double_free(void)
190 +{
191 + hashmap_free(&map, 1);
192 +}
193 +
194 +static struct long2double *find_entry(long key)
195 +{
196 + struct long2double k;
197 + hashmap_entry_init(&k, memhash(&key, sizeof(long)));
198 + k.key = key;
199 + return hashmap_get(&map, &k, NULL);
200 +}
201 +
202 +double get_value(long key)
203 +{
204 + struct long2double *e = find_entry(key);
205 + return e ? e->value : 0;
206 +}
207 +
208 +void set_value(long key, double value)
209 +{
210 + struct long2double *e = find_entry(key);
211 + if (!e) {
212 + e = malloc(sizeof(struct long2double));
213 + hashmap_entry_init(e, memhash(&key, sizeof(long)));
214 + e->key = key;
215 + hashmap_add(&map, e);
216 + }
217 + e->value = value;
218 +}
219 +------------
220 +
221 +Using variable-sized keys
222 +-------------------------
223 +
224 +The `hashmap_entry_get` and `hashmap_entry_remove` functions expect an ordinary
225 +`hashmap_entry` structure as key to find the correct entry. If the key data is
226 +variable-sized (e.g. a FLEX_ARRAY string) or quite large, it is undesirable
227 +to create a full-fledged entry structure on the heap and copy all the key data
228 +into the structure.
229 +
230 +In this case, the `keydata` parameter can be used to pass
231 +variable-sized key data directly to the comparison function, and the `key`
232 +parameter can be a stripped-down, fixed size entry structure allocated on the
233 +stack.
234 +
235 +See test-hashmap.c for an example using arbitrary-length strings as keys.
Makefile
+3
@@ -557,6 +557,7 @@ TEST_PROGRAMS_NEED_X += test-date
557 TEST_PROGRAMS_NEED_X += test-delta
558 TEST_PROGRAMS_NEED_X += test-dump-cache-tree
559 TEST_PROGRAMS_NEED_X += test-genrandom
560 +TEST_PROGRAMS_NEED_X += test-hashmap
561 TEST_PROGRAMS_NEED_X += test-index-version
562 TEST_PROGRAMS_NEED_X += test-line-buffer
563 TEST_PROGRAMS_NEED_X += test-match-trees
@@ -677,6 +678,7 @@ LIB_H += gpg-interface.h
678 LIB_H += graph.h
679 LIB_H += grep.h
680 LIB_H += hash.h
681 +LIB_H += hashmap.h
682 LIB_H += help.h
683 LIB_H += http.h
684 LIB_H += kwset.h
@@ -808,6 +810,7 @@ LIB_OBJS += gpg-interface.o
810 LIB_OBJS += graph.o
811 LIB_OBJS += grep.o
812 LIB_OBJS += hash.o
813 +LIB_OBJS += hashmap.o
814 LIB_OBJS += help.o
815 LIB_OBJS += hex.o
816 LIB_OBJS += ident.o
hashmap.c new
+228
@@ -0,0 +1,228 @@
1 +/*
2 + * Generic implementation of hash-based key value mappings.
3 + */
4 +#include "cache.h"
5 +#include "hashmap.h"
6 +
7 +#define FNV32_BASE ((unsigned int) 0x811c9dc5)
8 +#define FNV32_PRIME ((unsigned int) 0x01000193)
9 +
10 +unsigned int strhash(const char *str)
11 +{
12 + unsigned int c, hash = FNV32_BASE;
13 + while ((c = (unsigned char) *str++))
14 + hash = (hash * FNV32_PRIME) ^ c;
15 + return hash;
16 +}
17 +
18 +unsigned int strihash(const char *str)
19 +{
20 + unsigned int c, hash = FNV32_BASE;
21 + while ((c = (unsigned char) *str++)) {
22 + if (c >= 'a' && c <= 'z')
23 + c -= 'a' - 'A';
24 + hash = (hash * FNV32_PRIME) ^ c;
25 + }
26 + return hash;
27 +}
28 +
29 +unsigned int memhash(const void *buf, size_t len)
30 +{
31 + unsigned int hash = FNV32_BASE;
32 + unsigned char *ucbuf = (unsigned char *) buf;
33 + while (len--) {
34 + unsigned int c = *ucbuf++;
35 + hash = (hash * FNV32_PRIME) ^ c;
36 + }
37 + return hash;
38 +}
39 +
40 +unsigned int memihash(const void *buf, size_t len)
41 +{
42 + unsigned int hash = FNV32_BASE;
43 + unsigned char *ucbuf = (unsigned char *) buf;
44 + while (len--) {
45 + unsigned int c = *ucbuf++;
46 + if (c >= 'a' && c <= 'z')
47 + c -= 'a' - 'A';
48 + hash = (hash * FNV32_PRIME) ^ c;
49 + }
50 + return hash;
51 +}
52 +
53 +#define HASHMAP_INITIAL_SIZE 64
54 +/* grow / shrink by 2^2 */
55 +#define HASHMAP_RESIZE_BITS 2
56 +/* load factor in percent */
57 +#define HASHMAP_LOAD_FACTOR 80
58 +
59 +static void alloc_table(struct hashmap *map, unsigned int size)
60 +{
61 + map->tablesize = size;
62 + map->table = xcalloc(size, sizeof(struct hashmap_entry *));
63 +
64 + /* calculate resize thresholds for new size */
65 + map->grow_at = (unsigned int) ((uint64_t) size * HASHMAP_LOAD_FACTOR / 100);
66 + if (size <= HASHMAP_INITIAL_SIZE)
67 + map->shrink_at = 0;
68 + else
69 + /*
70 + * The shrink-threshold must be slightly smaller than
71 + * (grow-threshold / resize-factor) to prevent erratic resizing,
72 + * thus we divide by (resize-factor + 1).
73 + */
74 + map->shrink_at = map->grow_at / ((1 << HASHMAP_RESIZE_BITS) + 1);
75 +}
76 +
77 +static inline int entry_equals(const struct hashmap *map,
78 + const struct hashmap_entry *e1, const struct hashmap_entry *e2,
79 + const void *keydata)
80 +{
81 + return (e1 == e2) || (e1->hash == e2->hash && !map->cmpfn(e1, e2, keydata));
82 +}
83 +
84 +static inline unsigned int bucket(const struct hashmap *map,
85 + const struct hashmap_entry *key)
86 +{
87 + return key->hash & (map->tablesize - 1);
88 +}
89 +
90 +static void rehash(struct hashmap *map, unsigned int newsize)
91 +{
92 + unsigned int i, oldsize = map->tablesize;
93 + struct hashmap_entry **oldtable = map->table;
94 +
95 + alloc_table(map, newsize);
96 + for (i = 0; i < oldsize; i++) {
97 + struct hashmap_entry *e = oldtable[i];
98 + while (e) {
99 + struct hashmap_entry *next = e->next;
100 + unsigned int b = bucket(map, e);
101 + e->next = map->table[b];
102 + map->table[b] = e;
103 + e = next;
104 + }
105 + }
106 + free(oldtable);
107 +}
108 +
109 +static inline struct hashmap_entry **find_entry_ptr(const struct hashmap *map,
110 + const struct hashmap_entry *key, const void *keydata)
111 +{
112 + struct hashmap_entry **e = &map->table[bucket(map, key)];
113 + while (*e && !entry_equals(map, *e, key, keydata))
114 + e = &(*e)->next;
115 + return e;
116 +}
117 +
118 +static int always_equal(const void *unused1, const void *unused2, const void *unused3)
119 +{
120 + return 0;
121 +}
122 +
123 +void hashmap_init(struct hashmap *map, hashmap_cmp_fn equals_function,
124 + size_t initial_size)
125 +{
126 + unsigned int size = HASHMAP_INITIAL_SIZE;
127 + map->size = 0;
128 + map->cmpfn = equals_function ? equals_function : always_equal;
129 +
130 + /* calculate initial table size and allocate the table */
131 + initial_size = (unsigned int) ((uint64_t) initial_size * 100
132 + / HASHMAP_LOAD_FACTOR);
133 + while (initial_size > size)
134 + size <<= HASHMAP_RESIZE_BITS;
135 + alloc_table(map, size);
136 +}
137 +
138 +void hashmap_free(struct hashmap *map, int free_entries)
139 +{
140 + if (!map || !map->table)
141 + return;
142 + if (free_entries) {
143 + struct hashmap_iter iter;
144 + struct hashmap_entry *e;
145 + hashmap_iter_init(map, &iter);
146 + while ((e = hashmap_iter_next(&iter)))
147 + free(e);
148 + }
149 + free(map->table);
150 + memset(map, 0, sizeof(*map));
151 +}
152 +
153 +void *hashmap_get(const struct hashmap *map, const void *key, const void *keydata)
154 +{
155 + return *find_entry_ptr(map, key, keydata);
156 +}
157 +
158 +void *hashmap_get_next(const struct hashmap *map, const void *entry)
159 +{
160 + struct hashmap_entry *e = ((struct hashmap_entry *) entry)->next;
161 + for (; e; e = e->next)
162 + if (entry_equals(map, entry, e, NULL))
163 + return e;
164 + return NULL;
165 +}
166 +
167 +void hashmap_add(struct hashmap *map, void *entry)
168 +{
169 + unsigned int b = bucket(map, entry);
170 +
171 + /* add entry */
172 + ((struct hashmap_entry *) entry)->next = map->table[b];
173 + map->table[b] = entry;
174 +
175 + /* fix size and rehash if appropriate */
176 + map->size++;
177 + if (map->size > map->grow_at)
178 + rehash(map, map->tablesize << HASHMAP_RESIZE_BITS);
179 +}
180 +
181 +void *hashmap_remove(struct hashmap *map, const void *key, const void *keydata)
182 +{
183 + struct hashmap_entry *old;
184 + struct hashmap_entry **e = find_entry_ptr(map, key, keydata);
185 + if (!*e)
186 + return NULL;
187 +
188 + /* remove existing entry */
189 + old = *e;
190 + *e = old->next;
191 + old->next = NULL;
192 +
193 + /* fix size and rehash if appropriate */
194 + map->size--;
195 + if (map->size < map->shrink_at)
196 + rehash(map, map->tablesize >> HASHMAP_RESIZE_BITS);
197 + return old;
198 +}
199 +
200 +void *hashmap_put(struct hashmap *map, void *entry)
201 +{
202 + struct hashmap_entry *old = hashmap_remove(map, entry, NULL);
203 + hashmap_add(map, entry);
204 + return old;
205 +}
206 +
207 +void hashmap_iter_init(struct hashmap *map, struct hashmap_iter *iter)
208 +{
209 + iter->map = map;
210 + iter->tablepos = 0;
211 + iter->next = NULL;
212 +}
213 +
214 +void *hashmap_iter_next(struct hashmap_iter *iter)
215 +{
216 + struct hashmap_entry *current = iter->next;
217 + for (;;) {
218 + if (current) {
219 + iter->next = current->next;
220 + return current;
221 + }
222 +
223 + if (iter->tablepos >= iter->map->tablesize)
224 + return NULL;
225 +
226 + current = iter->map->table[iter->tablepos++];
227 + }
228 +}
hashmap.h new
+71
@@ -0,0 +1,71 @@
1 +#ifndef HASHMAP_H
2 +#define HASHMAP_H
3 +
4 +/*
5 + * Generic implementation of hash-based key-value mappings.
6 + * See Documentation/technical/api-hashmap.txt.
7 + */
8 +
9 +/* FNV-1 functions */
10 +
11 +extern unsigned int strhash(const char *buf);
12 +extern unsigned int strihash(const char *buf);
13 +extern unsigned int memhash(const void *buf, size_t len);
14 +extern unsigned int memihash(const void *buf, size_t len);
15 +
16 +/* data structures */
17 +
18 +struct hashmap_entry {
19 + struct hashmap_entry *next;
20 + unsigned int hash;
21 +};
22 +
23 +typedef int (*hashmap_cmp_fn)(const void *entry, const void *entry_or_key,
24 + const void *keydata);
25 +
26 +struct hashmap {
27 + struct hashmap_entry **table;
28 + hashmap_cmp_fn cmpfn;
29 + unsigned int size, tablesize, grow_at, shrink_at;
30 +};
31 +
32 +struct hashmap_iter {
33 + struct hashmap *map;
34 + struct hashmap_entry *next;
35 + unsigned int tablepos;
36 +};
37 +
38 +/* hashmap functions */
39 +
40 +extern void hashmap_init(struct hashmap *map, hashmap_cmp_fn equals_function,
41 + size_t initial_size);
42 +extern void hashmap_free(struct hashmap *map, int free_entries);
43 +
44 +/* hashmap_entry functions */
45 +
46 +static inline void hashmap_entry_init(void *entry, int hash)
47 +{
48 + struct hashmap_entry *e = entry;
49 + e->hash = hash;
50 + e->next = NULL;
51 +}
52 +extern void *hashmap_get(const struct hashmap *map, const void *key,
53 + const void *keydata);
54 +extern void *hashmap_get_next(const struct hashmap *map, const void *entry);
55 +extern void hashmap_add(struct hashmap *map, void *entry);
56 +extern void *hashmap_put(struct hashmap *map, void *entry);
57 +extern void *hashmap_remove(struct hashmap *map, const void *key,
58 + const void *keydata);
59 +
60 +/* hashmap_iter functions */
61 +
62 +extern void hashmap_iter_init(struct hashmap *map, struct hashmap_iter *iter);
63 +extern void *hashmap_iter_next(struct hashmap_iter *iter);
64 +static inline void *hashmap_iter_first(struct hashmap *map,
65 + struct hashmap_iter *iter)
66 +{
67 + hashmap_iter_init(map, iter);
68 + return hashmap_iter_next(iter);
69 +}
70 +
71 +#endif
t/t0011-hashmap.sh new
+240
@@ -0,0 +1,240 @@
1 +#!/bin/sh
2 +
3 +test_description='test hashmap and string hash functions'
4 +. ./test-lib.sh
5 +
6 +test_hashmap() {
7 + echo "$1" | test-hashmap $3 > actual &&
8 + echo "$2" > expect &&
9 + test_cmp expect actual
10 +}
11 +
12 +test_expect_success 'hash functions' '
13 +
14 +test_hashmap "hash key1" "2215982743 2215982743 116372151 116372151" &&
15 +test_hashmap "hash key2" "2215982740 2215982740 116372148 116372148" &&
16 +test_hashmap "hash fooBarFrotz" "1383912807 1383912807 3189766727 3189766727" &&
17 +test_hashmap "hash foobarfrotz" "2862305959 2862305959 3189766727 3189766727"
18 +
19 +'
20 +
21 +test_expect_success 'put' '
22 +
23 +test_hashmap "put key1 value1
24 +put key2 value2
25 +put fooBarFrotz value3
26 +put foobarfrotz value4
27 +size" "NULL
28 +NULL
29 +NULL
30 +NULL
31 +64 4"
32 +
33 +'
34 +
35 +test_expect_success 'put (case insensitive)' '
36 +
37 +test_hashmap "put key1 value1
38 +put key2 value2
39 +put fooBarFrotz value3
40 +size" "NULL
41 +NULL
42 +NULL
43 +64 3" ignorecase
44 +
45 +'
46 +
47 +test_expect_success 'replace' '
48 +
49 +test_hashmap "put key1 value1
50 +put key1 value2
51 +put fooBarFrotz value3
52 +put fooBarFrotz value4
53 +size" "NULL
54 +value1
55 +NULL
56 +value3
57 +64 2"
58 +
59 +'
60 +
61 +test_expect_success 'replace (case insensitive)' '
62 +
63 +test_hashmap "put key1 value1
64 +put Key1 value2
65 +put fooBarFrotz value3
66 +put foobarfrotz value4
67 +size" "NULL
68 +value1
69 +NULL
70 +value3
71 +64 2" ignorecase
72 +
73 +'
74 +
75 +test_expect_success 'get' '
76 +
77 +test_hashmap "put key1 value1
78 +put key2 value2
79 +put fooBarFrotz value3
80 +put foobarfrotz value4
81 +get key1
82 +get key2
83 +get fooBarFrotz
84 +get notInMap" "NULL
85 +NULL
86 +NULL
87 +NULL
88 +value1
89 +value2
90 +value3
91 +NULL"
92 +
93 +'
94 +
95 +test_expect_success 'get (case insensitive)' '
96 +
97 +test_hashmap "put key1 value1
98 +put key2 value2
99 +put fooBarFrotz value3
100 +get Key1
101 +get keY2
102 +get foobarfrotz
103 +get notInMap" "NULL
104 +NULL
105 +NULL
106 +value1
107 +value2
108 +value3
109 +NULL" ignorecase
110 +
111 +'
112 +
113 +test_expect_success 'add' '
114 +
115 +test_hashmap "add key1 value1
116 +add key1 value2
117 +add fooBarFrotz value3
118 +add fooBarFrotz value4
119 +get key1
120 +get fooBarFrotz
121 +get notInMap" "value2
122 +value1
123 +value4
124 +value3
125 +NULL"
126 +
127 +'
128 +
129 +test_expect_success 'add (case insensitive)' '
130 +
131 +test_hashmap "add key1 value1
132 +add Key1 value2
133 +add fooBarFrotz value3
134 +add foobarfrotz value4
135 +get key1
136 +get Foobarfrotz
137 +get notInMap" "value2
138 +value1
139 +value4
140 +value3
141 +NULL" ignorecase
142 +
143 +'
144 +
145 +test_expect_success 'remove' '
146 +
147 +test_hashmap "put key1 value1
148 +put key2 value2
149 +put fooBarFrotz value3
150 +remove key1
151 +remove key2
152 +remove notInMap
153 +size" "NULL
154 +NULL
155 +NULL
156 +value1
157 +value2
158 +NULL
159 +64 1"
160 +
161 +'
162 +
163 +test_expect_success 'remove (case insensitive)' '
164 +
165 +test_hashmap "put key1 value1
166 +put key2 value2
167 +put fooBarFrotz value3
168 +remove Key1
169 +remove keY2
170 +remove notInMap
171 +size" "NULL
172 +NULL
173 +NULL
174 +value1
175 +value2
176 +NULL
177 +64 1" ignorecase
178 +
179 +'
180 +
181 +test_expect_success 'iterate' '
182 +
183 +test_hashmap "put key1 value1
184 +put key2 value2
185 +put fooBarFrotz value3
186 +iterate" "NULL
187 +NULL
188 +NULL
189 +key2 value2
190 +key1 value1
191 +fooBarFrotz value3"
192 +
193 +'
194 +
195 +test_expect_success 'iterate (case insensitive)' '
196 +
197 +test_hashmap "put key1 value1
198 +put key2 value2
199 +put fooBarFrotz value3
200 +iterate" "NULL
201 +NULL
202 +NULL
203 +fooBarFrotz value3
204 +key2 value2
205 +key1 value1" ignorecase
206 +
207 +'
208 +
209 +test_expect_success 'grow / shrink' '
210 +
211 + rm -f in &&
212 + rm -f expect &&
213 + for n in $(test_seq 51)
214 + do
215 + echo put key$n value$n >> in &&
216 + echo NULL >> expect
217 + done &&
218 + echo size >> in &&
219 + echo 64 51 >> expect &&
220 + echo put key52 value52 >> in &&
221 + echo NULL >> expect
222 + echo size >> in &&
223 + echo 256 52 >> expect &&
224 + for n in $(test_seq 12)
225 + do
226 + echo remove key$n >> in &&
227 + echo value$n >> expect
228 + done &&
229 + echo size >> in &&
230 + echo 256 40 >> expect &&
231 + echo remove key40 >> in &&
232 + echo value40 >> expect &&
233 + echo size >> in &&
234 + echo 64 39 >> expect &&
235 + cat in | test-hashmap > out &&
236 + test_cmp expect out
237 +
238 +'
239 +
240 +test_done
test-hashmap.c new
+340
@@ -0,0 +1,340 @@
1 +#include "cache.h"
2 +#include "hashmap.h"
3 +#include <stdio.h>
4 +
5 +struct test_entry
6 +{
7 + struct hashmap_entry ent;
8 + /* key and value as two \0-terminated strings */
9 + char key[FLEX_ARRAY];
10 +};
11 +
12 +static const char *get_value(const struct test_entry *e)
13 +{
14 + return e->key + strlen(e->key) + 1;
15 +}
16 +
17 +static int test_entry_cmp(const struct test_entry *e1,
18 + const struct test_entry *e2, const char* key)
19 +{
20 + return strcmp(e1->key, key ? key : e2->key);
21 +}
22 +
23 +static int test_entry_cmp_icase(const struct test_entry *e1,
24 + const struct test_entry *e2, const char* key)
25 +{
26 + return strcasecmp(e1->key, key ? key : e2->key);
27 +}
28 +
29 +static struct test_entry *alloc_test_entry(int hash, char *key, int klen,
30 + char *value, int vlen)
31 +{
32 + struct test_entry *entry = malloc(sizeof(struct test_entry) + klen
33 + + vlen + 2);
34 + hashmap_entry_init(entry, hash);
35 + memcpy(entry->key, key, klen + 1);
36 + memcpy(entry->key + klen + 1, value, vlen + 1);
37 + return entry;
38 +}
39 +
40 +#define HASH_METHOD_FNV 0
41 +#define HASH_METHOD_I 1
42 +#define HASH_METHOD_IDIV10 2
43 +#define HASH_METHOD_0 3
44 +#define HASH_METHOD_X2 4
45 +#define TEST_SPARSE 8
46 +#define TEST_ADD 16
47 +#define TEST_SIZE 100000
48 +
49 +static unsigned int hash(unsigned int method, unsigned int i, const char *key)
50 +{
51 + unsigned int hash;
52 + switch (method & 3)
53 + {
54 + case HASH_METHOD_FNV:
55 + hash = strhash(key);
56 + break;
57 + case HASH_METHOD_I:
58 + hash = i;
59 + break;
60 + case HASH_METHOD_IDIV10:
61 + hash = i / 10;
62 + break;
63 + case HASH_METHOD_0:
64 + hash = 0;
65 + break;
66 + }
67 +
68 + if (method & HASH_METHOD_X2)
69 + hash = 2 * hash;
70 + return hash;
71 +}
72 +
73 +/*
74 + * Test performance of hashmap.[ch]
75 + * Usage: time echo "perfhashmap method rounds" | test-hashmap
76 + */
77 +static void perf_hashmap(unsigned int method, unsigned int rounds)
78 +{
79 + struct hashmap map;
80 + char buf[16];
81 + struct test_entry **entries;
82 + unsigned int *hashes;
83 + unsigned int i, j;
84 +
85 + entries = malloc(TEST_SIZE * sizeof(struct test_entry *));
86 + hashes = malloc(TEST_SIZE * sizeof(int));
87 + for (i = 0; i < TEST_SIZE; i++) {
88 + snprintf(buf, sizeof(buf), "%i", i);
89 + entries[i] = alloc_test_entry(0, buf, strlen(buf), "", 0);
90 + hashes[i] = hash(method, i, entries[i]->key);
91 + }
92 +
93 + if (method & TEST_ADD) {
94 + /* test adding to the map */
95 + for (j = 0; j < rounds; j++) {
96 + hashmap_init(&map, (hashmap_cmp_fn) test_entry_cmp, 0);
97 +
98 + /* add entries */
99 + for (i = 0; i < TEST_SIZE; i++) {
100 + hashmap_entry_init(entries[i], hashes[i]);
101 + hashmap_add(&map, entries[i]);
102 + }
103 +
104 + hashmap_free(&map, 0);
105 + }
106 + } else {
107 + /* test map lookups */
108 + hashmap_init(&map, (hashmap_cmp_fn) test_entry_cmp, 0);
109 +
110 + /* fill the map (sparsely if specified) */
111 + j = (method & TEST_SPARSE) ? TEST_SIZE / 10 : TEST_SIZE;
112 + for (i = 0; i < j; i++) {
113 + hashmap_entry_init(entries[i], hashes[i]);
114 + hashmap_add(&map, entries[i]);
115 + }
116 +
117 + for (j = 0; j < rounds; j++) {
118 + for (i = 0; i < TEST_SIZE; i++) {
119 + struct hashmap_entry key;
120 + hashmap_entry_init(&key, hashes[i]);
121 + hashmap_get(&map, &key, entries[i]->key);
122 + }
123 + }
124 +
125 + hashmap_free(&map, 0);
126 + }
127 +}
128 +
129 +struct hash_entry
130 +{
131 + struct hash_entry *next;
132 + char key[FLEX_ARRAY];
133 +};
134 +
135 +/*
136 + * Test performance of hash.[ch]
137 + * Usage: time echo "perfhash method rounds" | test-hashmap
138 + */
139 +static void perf_hash(unsigned int method, unsigned int rounds)
140 +{
141 + struct hash_table map;
142 + char buf[16];
143 + struct hash_entry **entries, **res, *entry;
144 + unsigned int *hashes;
145 + unsigned int i, j;
146 +
147 + entries = malloc(TEST_SIZE * sizeof(struct hash_entry *));
148 + hashes = malloc(TEST_SIZE * sizeof(int));
149 + for (i = 0; i < TEST_SIZE; i++) {
150 + snprintf(buf, sizeof(buf), "%i", i);
151 + entries[i] = malloc(sizeof(struct hash_entry) + strlen(buf) + 1);
152 + strcpy(entries[i]->key, buf);
153 + hashes[i] = hash(method, i, entries[i]->key);
154 + }
155 +
156 + if (method & TEST_ADD) {
157 + /* test adding to the map */
158 + for (j = 0; j < rounds; j++) {
159 + init_hash(&map);
160 +
161 + /* add entries */
162 + for (i = 0; i < TEST_SIZE; i++) {
163 + res = (struct hash_entry **) insert_hash(
164 + hashes[i], entries[i], &map);
165 + if (res) {
166 + entries[i]->next = *res;
167 + *res = entries[i];
168 + } else {
169 + entries[i]->next = NULL;
170 + }
171 + }
172 +
173 + free_hash(&map);
174 + }
175 + } else {
176 + /* test map lookups */
177 + init_hash(&map);
178 +
179 + /* fill the map (sparsely if specified) */
180 + j = (method & TEST_SPARSE) ? TEST_SIZE / 10 : TEST_SIZE;
181 + for (i = 0; i < j; i++) {
182 + res = (struct hash_entry **) insert_hash(hashes[i],
183 + entries[i], &map);
184 + if (res) {
185 + entries[i]->next = *res;
186 + *res = entries[i];
187 + } else {
188 + entries[i]->next = NULL;
189 + }
190 + }
191 +
192 + for (j = 0; j < rounds; j++) {
193 + for (i = 0; i < TEST_SIZE; i++) {
194 + entry = lookup_hash(hashes[i], &map);
195 + while (entry) {
196 + if (!strcmp(entries[i]->key, entry->key))
197 + break;
198 + entry = entry->next;
199 + }
200 + }
201 + }
202 +
203 + free_hash(&map);
204 +
205 + }
206 +}
207 +
208 +#define DELIM " \t\r\n"
209 +
210 +/*
211 + * Read stdin line by line and print result of commands to stdout:
212 + *
213 + * hash key -> strhash(key) memhash(key) strihash(key) memihash(key)
214 + * put key value -> NULL / old value
215 + * get key -> NULL / value
216 + * remove key -> NULL / old value
217 + * iterate -> key1 value1\nkey2 value2\n...
218 + * size -> tablesize numentries
219 + *
220 + * perfhashmap method rounds -> test hashmap.[ch] performance
221 + * perfhash method rounds -> test hash.[ch] performance
222 + */
223 +int main(int argc, char *argv[])
224 +{
225 + char line[1024];
226 + struct hashmap map;
227 + int icase;
228 +
229 + /* init hash map */
230 + icase = argc > 1 && !strcmp("ignorecase", argv[1]);
231 + hashmap_init(&map, (hashmap_cmp_fn) (icase ? test_entry_cmp_icase
232 + : test_entry_cmp), 0);
233 +
234 + /* process commands from stdin */
235 + while (fgets(line, sizeof(line), stdin)) {
236 + char *cmd, *p1 = NULL, *p2 = NULL;
237 + int l1 = 0, l2 = 0, hash = 0;
238 + struct test_entry *entry;
239 +
240 + /* break line into command and up to two parameters */
241 + cmd = strtok(line, DELIM);
242 + /* ignore empty lines */
243 + if (!cmd || *cmd == '#')
244 + continue;
245 +
246 + p1 = strtok(NULL, DELIM);
247 + if (p1) {
248 + l1 = strlen(p1);
249 + hash = icase ? strihash(p1) : strhash(p1);
250 + p2 = strtok(NULL, DELIM);
251 + if (p2)
252 + l2 = strlen(p2);
253 + }
254 +
255 + if (!strcmp("hash", cmd) && l1) {
256 +
257 + /* print results of different hash functions */
258 + printf("%u %u %u %u\n", strhash(p1), memhash(p1, l1),
259 + strihash(p1), memihash(p1, l1));
260 +
261 + } else if (!strcmp("add", cmd) && l1 && l2) {
262 +
263 + /* create entry with key = p1, value = p2 */
264 + entry = alloc_test_entry(hash, p1, l1, p2, l2);
265 +
266 + /* add to hashmap */
267 + hashmap_add(&map, entry);
268 +
269 + } else if (!strcmp("put", cmd) && l1 && l2) {
270 +
271 + /* create entry with key = p1, value = p2 */
272 + entry = alloc_test_entry(hash, p1, l1, p2, l2);
273 +
274 + /* add / replace entry */
275 + entry = hashmap_put(&map, entry);
276 +
277 + /* print and free replaced entry, if any */
278 + puts(entry ? get_value(entry) : "NULL");
279 + free(entry);
280 +
281 + } else if (!strcmp("get", cmd) && l1) {
282 +
283 + /* setup static key */
284 + struct hashmap_entry key;
285 + hashmap_entry_init(&key, hash);
286 +
287 + /* lookup entry in hashmap */
288 + entry = hashmap_get(&map, &key, p1);
289 +
290 + /* print result */
291 + if (!entry)
292 + puts("NULL");
293 + while (entry) {
294 + puts(get_value(entry));
295 + entry = hashmap_get_next(&map, entry);
296 + }
297 +
298 + } else if (!strcmp("remove", cmd) && l1) {
299 +
300 + /* setup static key */
301 + struct hashmap_entry key;
302 + hashmap_entry_init(&key, hash);
303 +
304 + /* remove entry from hashmap */
305 + entry = hashmap_remove(&map, &key, p1);
306 +
307 + /* print result and free entry*/
308 + puts(entry ? get_value(entry) : "NULL");
309 + free(entry);
310 +
311 + } else if (!strcmp("iterate", cmd)) {
312 +
313 + struct hashmap_iter iter;
314 + hashmap_iter_init(&map, &iter);
315 + while ((entry = hashmap_iter_next(&iter)))
316 + printf("%s %s\n", entry->key, get_value(entry));
317 +
318 + } else if (!strcmp("size", cmd)) {
319 +
320 + /* print table sizes */
321 + printf("%u %u\n", map.tablesize, map.size);
322 +
323 + } else if (!strcmp("perfhashmap", cmd) && l1 && l2) {
324 +
325 + perf_hashmap(atoi(p1), atoi(p2));
326 +
327 + } else if (!strcmp("perfhash", cmd) && l1 && l2) {
328 +
329 + perf_hash(atoi(p1), atoi(p2));
330 +
331 + } else {
332 +
333 + printf("Unknown command %s\n", cmd);
334 +
335 + }
336 + }
337 +
338 + hashmap_free(&map, 1);
339 + return 0;
340 +}