(I'm not good at English. So please proofread this document and point out mistakes to me.)
QDBM is a library of routines for managing a database. The database is a simple data file containing records, each is a pair of a key and a value. Every key and value is serial bytes with variable length. Both binary data and character string can be used as a key and a value. There is neither concept of data tables nor data types. Each key must be unique within a database, so it is impossible to store two or more records with a key overlaps.
The following access methods are provided to the database: storing a record with a key and a value, deleting a record by a key, retrieving a record by a key. Moreover, traversal access to every key are provided, but the order is arbitrary. These access methods are similar to ones of DBM (or its compatibles, NDBM and GDBM) library defined in the UNIX standard. QDBM is an alternative for DBM because of its higher performance.
QDBM is developed referring to GDBM for the purpose of the following three points: higher processing speed, smaller size of a database file, and simpler API. The features of QDBM are just having attained them. Moreover, the following three restrictions of DBM: a process can handle only one database, the size of a key and a value is bounded, a database file is sparse, are cleared.
QDBM uses hash algorithm to retrieve records. If a bucket array has sufficient number of elements, the time complexity of retrieval is `O(1)'. That is, time required for retrieving a record is constant, regardless of the scale of the database. It is also the same about storing and deleting. Collision of hash values is managed by separate chain method. Data structure of the chain is binary search tree. Even if a bucket array has unusually scarce elements, the time complexity of retrieval is `O(log n)'.
QDBM attains improvement in retrieval by loading whole of a bucket array onto RAM. If a bucket array is on RAM, it is possible to access a region of a target record by about one pass of file operations. A bucket array saved in a file is not read into RAM with the `read' call but directly mapped to RAM with the `mmap' call. Therefore, preparation time on connecting to a database is very short, and two or more processes can share the same memory map.
If number of elements of a bucket array is about half of records stored within a database, although it depends on characteristic of the input, the rate of collision of hash values are about 56.7% (36.8% if the same, 21.3% if twice, 11.5% if four times, 6.0% if eight times). In such case, it is possible to retrieve a record by two or less passes of file operations. If it is made into a performance index, in order to handle a database containing one million of records, a bucket array with a half million of elements is needed. The size of each element is 4 bytes. That is, if 2M bytes of RAM is available, a database containing one million records can be handled.
QDBM has two kinds of methods to connect to a database. A `reader' can perform retrieving but neither storing nor deleting. A `writer' can perform all access methods. Exclusion control between processes is performed when connecting to a database by file locking. While a writer is connecting to a database, neither readers nor writers can connect. While a reader is connecting to a database, other readers can connect, but neither writers can. According to this mechanism, the adjustment of simultaneous connections to a database is guaranteed in multitasking environment. However, it does not correspond in multithread environment.
DBM has two modes of storing operation, `insert' and `replace'. In case a key overlaps an existing record, the insert mode keeps the existing value, while the replace mode transposes it to the specified value. In addition to the two modes, QDBM has `concatenate' mode. In the mode, the specified value is concatenated at the end of the existing value and stored. This feature is useful when adding a element to a value as an array. Moreover, although DBM has a method to fetch out a value from a database only by reading a whole region of a record, QDBM has a method to fetch out a part of a region of a value. When a value is treated as an array, this feature is also useful.
If data alignment is assigned to a database, each record will place in a file with vacating suitable padding bytes. When it is going to overwrite a value of larger size than the size of the existing value or to concatenate a specified value to a existing value, if increasing size is settled in size of the padding, it is not necessary to move the region of the existing record to another position. Although the processing which moves the region of a record requires amount of calculation according to size of the region, the performance of updating database to the scale of stored records is kept constant by taking padding in size according to size of each record.
QDBM has three kinds of APIs: the basic API, the extended API and the compatible API. In the basic API, a database is treated as a file. In the extended API, a database is treated as a directory containing one or more database files. If it is going to store all the data of a database in one file, file size may exceed restriction of a file system. In the extended API, a database is divided into two or more files in a directory. Since the basic API and the extended API are resembled mutually, it is easy to porting an application between each API. The compatible API is for porting an application from NDBM to QDBM. Besides, QDBM has command line interfaces corresponding to the three APIs. They are useful for unit test, debugging applications and so on. They will also support prototyping of database applications with script languages.
To install QDBM, `gcc' and `make' are required.
When an archive file of QDBM is extracted, change the current working directory to the generated directory and perform installation.
Run the configuration script.
./configure
Build programs.
make
Perform self-diagnostic test.
make check
Install programs. This operation must be carried out by the root user.
make install
When a series of work finishes, header files, `depot.h', `curia.h' and `relic.h' will be installed in `/usr/local/include', libraries, `libqdbm.a' and `libqdbm.so' will be installed in `/usr/local/lib', executable commands, `dpmgr', `dptest', `dptsv', `crmgr', `crtest', `rlmgr' and `rltest' will be installed in `/usr/local/bin'. Besides, on Windows (Cygwin), libraries and its destinations of installation are differ from ones of generic UNIX environments. `libqdbm.a' and `libqdbm.dll.a' will be installed in `/usr/local/lib'. `qdbm.dll' will be installed in `$SYSTEMROOT/system32'.
To uninstall QDBM, execute the following command after `./configure'. This operation must be carried out by the root user.
make uninstall
If an old version of QDBM is installed on your system, uninstall it before installation of a new version.
Depot is the basic API of QDBM. In order to use Depot, you should include `depot.h' and `stdlib.h' in source code. Usually, the following description will be near the beginning of a source file.
A pointer to DEPOT is used as a database handle. It is like that some file I/O routines of `stdio.h' use a pointer to FILE. A database handle is opened with the function `dpopen' and closed with `dpclose'. You should not refer directly to any member of a handle. If a fatal error occurs in a database, any access method via the handle except `dpclose' will not work and return error status.
You can assign a file descriptor for debugging information to the external variable `dpdbgfd'. The initial value is -1. If the value is positive, some debugging information is output into the file descriptor.
A status of the last happened error is assigned to the external variable `dpecode'. The initial value is is DP_ENOERR. Refer to `depot.h' for details of the type DPECODE.
You can use the function `dperrmsg' in order to obtain a message string corresponding to an error code.
You can use the function `dpopen' in order to obtain a database handle. Size of a bucket array of a database is determined on creating, and is not to be changed except for by optimization of the database. Suggested size of a bucket array is about from 0.5 to 4 times of number of all records to store. While connecting as a writer, an exclusive lock is invoked to the database. While connecting as a reader, a shared lock is invoked to the database. Control blocks until the lock is achieved.
Every connected handle is to be closed with the function `dpclose'. Contents of updating of a database is synchronized with a file when closing connection. When a writer opens a database, a database will be destroyed if it does not close appropriately.
You can use the function `dpput' in order to store a recored.
You can use the function `dpout' in order to delete a recored.
You can use the function `dpget' in order to retrieve a recored. The region of the return value is allocated with the `malloc' call, so is to be released with the `free' call.
You can use the function `dpvsiz' in order to know length of a value of a recored. The function is faster than `dpget'.
You can use the function `dpiterinit' in order to initialize a iterator of a database handle for traversal access to every key in the database.
You can use the function `dpiternext' in order to get the next key of a iterator. Although the order cannot be controlled, every record in a database can be referred to. The region of the return value is allocated with the `malloc' call, so is to be released with the `free' call.
You can use the function `dpsetalign' in order to set alignment of a database handle. If alignment is assigned in a database handle, processing speed of overwriting an existing record goes up. Basic size of alignment is suggested to be average length of values of records to be stored. Unit size of alignment is suggested to be about from 4 times to 16 times of basic size.
You can use the function `dpsync' in order to synchronize contents of updating with a file and device. The function is useful when another process uses a connected database file.
You can use the function `dpoptimize' in order to optimize a database. In an alternating succession of storing in overwrite mode or concatenation mode, size of a database file comes big due to accumulated dispensable spaces. Such dispensable spaces are deleted by optimization.
You can use the function `dpname' in order to know a name of a database handle. The region of the return value is allocated with the `malloc' call, so is to be released with the `free' call.
You can use the function `dpname' in order to know size of a database file.
You can use the function `dpbnum' in order to know number of elements which a bucket array of a database has.
You can use the function `dpbnum' in order to know number of used elements of a bucket array.
You can use the function `dpbnum' in order to know number of records stored in a database.
You can use the function `dpwritable' in order to know whether a database handle is a writer or not.
You can use the function `dpfdesc' in order to get a file descriptor of a database file. However, handling a database file directly is not suggested.
The function `dpinnerhash' is a hash function used inside of Depot. The function is useful to know which element a key is to be stored into.
The function `dpouterhash' is a hash function without reference to hash functions used inside of Depot. The function is useful for hash algorithm used in an application of Depot.
The function `dpprimenum' is used in order to obtain a prime number not less than a number. The function is useful when an application determine size of a bucket array of its hash algorithm.
For building a program using Depot, the program should be linked with a library file `libqdbm.a' or `libqdbm.so'. For example, the following command is executed to build `hoge' from `hoge.c'.
gcc -I/usr/local/include -L/usr/local/lib -o hoge hoge.c -lqdbm
The following example stores and retrieves a phone number, using the name as the key.
#include <depot.h> #include <stdlib.h> #include <stdio.h> #define NAME "mikio" #define NUMBER "000-1234-5678" #define DBNAME "book" /* function prototypes */ int main(int argc, char **argv); /* main routine */ int main(int argc, char **argv){ DEPOT *depot; char *val; /* open the database */ if(!(depot = dpopen(DBNAME, DP_OWRITER | DP_OCREAT, -1))){ fprintf(stderr, "%s\n", dperrmsg(dpecode)); exit(1); } /* store the record */ if(!dpput(depot, NAME, -1, NUMBER, -1, DP_DOVER)){ fprintf(stderr, "%s\n", dperrmsg(dpecode)); } /* retrieve the record */ if(!(val = dpget(depot, NAME, -1, 0, -1, NULL))){ fprintf(stderr, "%s\n", dperrmsg(dpecode)); } else { printf("Name: %s\n", NAME); printf("Number: %s\n", val); free(val); } /* close the database */ if(!dpclose(depot)){ fprintf(stderr, "%s\n", dperrmsg(dpecode)); exit(1); } return 0; }
Depot has the following command line interfaces.
`dpmgr' is a utility for debugging Depot and its applications. It features editing and checking of a database. It can be use for database applications with shell script. `dpmgr' is used in the following format. `name' specifies a database name. `key' specifies a key of a record. `val' specifies a value of a record.
`dpmgr' returns 0 on success, another on failure.
`dptest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. `dptest' is used in the following format. `name' specifies a database name. `rnum' specifies a number of records. `bnum' specifies a number of elements of a bucket array.
`dptest' returns 0 on success, another on failure.
`dptsv' features mutual coversion between database of Depot and TSV text. `dptsv' is used in the following format. `name' specifies a database name. `export' reads TSV data from the standard input. If a key overlaps, the latter is adopted. `-bnum' specifies a number of elements of a bucket array. `import' writes TSV data to the standard output.
`dptsv' returns 0 on success, another on failure.
Curia is the extended API of QDBM. Curia provides routines for managing two or more database files in a directory. Restriction of some file systems that size of each file is limited is escaped by dividing a database file into two or more. Moreover, deployment of each database file on individual devices improves scalability.
In order to use Curia, you should include `depot.h', `curia.h' and `stdlib.h' in source code. Usually, the following description will be near the beginning of a source file.
A pointer to CURIA is used as a database handle. A database handle is opened with the function `cropen' and closed with `crclose'. You should not refer directly to any member of a handle.
You can use the function `cropen' in order to obtain a database handle. Size of a bucket array of a database is determined on creating, and is not to be changed except for by optimization of the database. Suggested size of a bucket array is about from 0.5 to 4 times of number of all records to store. Max number of division of databases is 256. While connecting as a writer, an exclusive lock is invoked to the database. While connecting as a reader, a shared lock is invoked to the database. Control blocks until the lock is achieved.
Every connected handle is to be closed with the function `crclose'. Contents of updating of a database is synchronized with a file when closing connection. When a writer opens a database, a database will be destroyed if it does not close appropriately.
The following routines are to be used as well as ones of Depot. Besides, there is no function corresponding to `dperrmsg', `dpinnerhash', `dpouterhash' and `dpprimenum'.
Curia has the following functions for managing large objects. While usual records are saved in a few database files, large objects are saved in defferent files for each other. Because files of large objects are deployed in directories by hash value, speed of retrieving is not so slow.
You can use the function `crputlob' in order to store a large object.
You can use the function `croutlob' in order to delete a large object.
You can use the function `crgetlob' in order to retrieve a large object. The region of the return value is allocated with the `malloc' call, so is to be released with the `free' call.
How to build programs using Curia is the same as the case of Depot.
Curia has the following command line interfaces.
`crmgr' is a utility for debugging Depot and its applications. It features editing and checking of a database. It can be use for database applications with shell script. `crmgr' is used in the following format. `name' specifies a database name. `key' specifies a key of a record. `val' specifies a value of a record.
`crmgr' returns 0 on success, another on failure.
`crtest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. `crtest' is used in the following format. `name' specifies a database name. `rnum' specifies a number of records. `bnum' specifies a number of elements of a bucket array.
`crtest' returns 0 on success, another on failure.
Relic is the compatible API with NDBM. So, Relic wraps Depot as API of NDBM. It is easy to port an application from NDBM to QDBM. In most cases, you should only replace includings of `ndbm.h' with `relic.h'.
The original NDBM treats a database as a pair of two files. One has a name with suffix `.dir' and stores a bit map of keys. The other has a name with suffix `.pag' and stores entities of all data. Relic creates a file of `.dir' as a mere dummy file and creates a file of `.pag' as a database. Relic has no restriction about length of data of each record. Relic cannot handle database files made by the original NDBM.
In order to use Relic, you should include `relic.h', `sys/types.h', `sys/stat.h' and `fcntl.h' in source code. Usually, the following description will be near the beginning of a source file.
A pointer to DBM is used as a database handle. A database handle is opened with the function `dbm_open' and closed with `dbm_close'. You should not refer directly to any member of a handle.
Structures of `datum' type is used in order to give and receive data of keys and values with functions of Relic.
You can use the function `dbm_open' in order to obtain a database handle.
Every connected handle is to be closed with the function `dbm_close'.
You can use the function `dbm_store' in order to store a recored.
You can use the function `dbm_delete' in order to delete a recored.
You can use the function `dbm_fetch' in order to retrieve a recored.
You can use the function `dbm_firstkey' in order to get the first key of a database. However, order of records is arbitrary.
You can use the function `dbm_nextkey' in order to get the next key of a database. However, order of records is arbitrary.
The function `dbm_error' has no effect.
The function `dbm_clearerr' has no effect.
You can use the function `dbm_rdonly' in order to check whether a handle is read-only or not.
You can use the function `dbm_dirfno' in order to get a file descriptor of a directory file. However, handling a database file directly is not suggested.
You can use the function `dbm_pagfno' in order to get a file descriptor of a data file. However, handling a database file directly is not suggested.
How to build programs using Relic is the same as the case of Depot. Note that an option to be given to a linker is not `-lndbm', but `-lqdbm'.
Relic has the following command line interfaces.
`rlmgr' is a utility for debugging Relic and its applications. It features editing and checking of a database. It can be use for database applications with shell script. `rlmgr' is used in the following format. `name' specifies a database name. `key' specifies a key of a record. `val' specifies a value of a record.
`rlmgr' returns 0 on success, another on failure.
`rltest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. `rltest' is used in the following format. `name' specifies a database name. `rnum' specifies a number of records.
`rltest' returns 0 on success, another on failure.
There is no bug which are found but not fixed. If you find any bug, report it to the author.
Database files are vulnerable for I/O error. QDBM has neither feature for rollback nor backup. Applications are responsible for handling such errors as disk-full and killing interruptive signals.
This program is written by Mikio Hirabayashi and distributed as a free software. You can redistribute it and/or modify it under the terms of the GNU General Public License Version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
You may contact the author by e-mail to <mikio@24h.co.jp>. Any suggestion or bug report is welcome to the author.