Tokyo Cabinet and Tokyo Tyrant
1. Tokyo Products
2. Tokyo Cabinet (a modern implementation of DBM)
2.1 Tokyo Cabinet 이란?
- Tokyo Cabinet은 hash table, B+ tree, fixed-length array 방식을 지원하는 DBM 스타일의 Embedded library 이다.
- Key-Value 쌍의 레코드를 저장하는 간단한 데이터 파일이다.
- data table이나 data types 개념도 있다.
- 비슷한 키-밸류 형태의 디비 : DBM, NDBM(New DBM), GDBM(GNU DBM), TDB, CDB, Berkeley DB
- SQL개념의 질의문 지원 안하고 API를 이용
- DBMS가 아니다.(library형태이니)
- Tokyo Cabinet is written in the C language
- provided as API of C, Perl, Ruby, Java, and Lua.
- Tokyo Cabinet is a free software licensed under the GNU Lesser General Public License.
2.2 Tokyo Cabinet의 특징
- high performance
- insert: 0.4 sec/1M records - 2,500,000 qps(Queries per second)
- search: 0.33 sec/1M records - 3,000,000 qps
- high concurrency
- multi-thread safe
- read/write locking by records
- high scalability
- hash and B+tree structure = O(1) and O(log N)
- no actual limit size of a database file (to 8 exabytes)
2.3 주요기능
TCHDB: Hash Database
- each key must be unique within a database (key가 unique해야 함)
- impossible to store two or more records with a key overlaps (중복키로 두개 이상 저장 불가)
- These access methods are similar to ones of DBM (or its followers: NDBM and GDBM) library defined in the UNIX standard
- static hashing : O(1) time complexity
TCBDB: B+ Tree Database
- As for database of B+ tree, records whose keys are duplicated can be stored (Key 중복된 레코드 저장 가능)
- Access methods of storing, deleting, and retrieving are provided as with the database of hash table(저장, 삭제, 검색방식은 해시 테이블의 데이터베이스로 제공된다.)
- It is possible to access each record with the cursor in ascending or descending order
- B+ tree : O(log N) time complexity
- custom comparison function : prefix/range matching
- cursor : jump/next/prev
TCFDB: Fixed-length Database
- As for database of fixed-length array, records are stored with unique natural numbers (레코드 저장시 unique natural number가 저장됨)
- It is impossible to store two or more records with a key overlaps (키 중복으로 두 개 이상의 레코드를 저장 불가)
- Moreover, the length of each record is limited by the specified length (또한, 각 레코드의 길이가 특정 길이로 제한)
- Provided operations are the same as ones of hash database.
- array of fixedlength elements : O(1) time complexity
TCTDB: Table Database
- Table database is also provided as a variant of hash database. (해시 데이터베이스를 변형으로 제공)
- Each record is identified by the primary key and has a set of named columns (각 레코드는 기본 키에 의해 식별, 이름 컬럼 세트 구조)
- Although there is no concept of data schema, it is possible to search for records with complex conditions efficiently by using indices of arbitrary columns.
(데이터 스키마의 개념이없지만, 효율적으로 임의의 열 인덱스를 사용하여 복잡한 조건에 레코드를 검색 할 수 있다.)
2.4 Tokyo Cabinet 설치
다운로드
설치
- tokyocabinet-1.4.48.tar.gz 파일을 다운받아서 configure, make, make install만 하면 된다.
- 아래 라이브러리가 필요하다. 없으면 모두 설치 해야 함
- zlib : for loss-less data compression. 1.2.3 or later is suggested.
- bzip2 : for loss-less data compression. 1.0.5 or later is suggested.
wget http://fallabs.com/tokyocabinet/tokyocabinet-1.4.48.tar.gz
tar zxvf tokyocabinet-1.4.48.tar.gz
cd tokyocabinet-1.4.48
./configure --prefix=/cloud/env/tokyocabinet
make
#================================================================
# Ready to install.
#================================================================
make install
#================================================================
# Thanks for using Tokyo Cabinet.
#================================================================
2.5 Tokyo Cabinet Example code
C Example code
*tokyocabinet-1.4.48.tar.gz 압축을 해제하면 example 폴더에 예제 파일이 존재함
include <tcutil.h>
#include <tchdb.h>
#include <stdlib.h>
#include <stdbool.h>
#include <stdint.h>
int main(int argc, char **argv){
TCHDB *hdb;
int ecode;
char *key, *value;
/* create the object */
hdb = tchdbnew();
/* open the database */
if(!tchdbopen(hdb, "casket.tch", HDBOWRITER | HDBOCREAT)){
ecode = tchdbecode(hdb);
fprintf(stderr, "open error: %s\n", tchdberrmsg(ecode));
}
/* store records */
if(!tchdbput2(hdb, "foo", "hop") ||
!tchdbput2(hdb, "bar", "step") ||
!tchdbput2(hdb, "baz", "jump")){
ecode = tchdbecode(hdb);
fprintf(stderr, "put error: %s\n", tchdberrmsg(ecode));
}
/* retrieve records */
value = tchdbget2(hdb, "foo");
if(value){
printf("%s\n", value);
free(value);
} else {
ecode = tchdbecode(hdb);
fprintf(stderr, "get error: %s\n", tchdberrmsg(ecode));
}
/* traverse records */
tchdbiterinit(hdb);
while((key = tchdbiternext2(hdb)) != NULL){
value = tchdbget2(hdb, key);
if(value){
printf("%s:%s\n", key, value);
free(value);
}
free(key);
}
/* close the database */
if(!tchdbclose(hdb)){
ecode = tchdbecode(hdb);
fprintf(stderr, "close error: %s\n", tchdberrmsg(ecode));
}
/* delete the object */
tchdbdel(hdb);
return 0;
}
JAVA Example code
import java.util.Map;
import java.util.HashMap;
import tokyocabinet.TDB;
public class TestTokyoCabinet {
public static void main(String[] args) {
TDB tdb = new TDB();
if(tdb.open("/tmp/tdb.db", TDB.OWRITER | TDB.OCREAT)) {
System.out.println("open successful");
} else {
System.out.println("open fail");
}
Map<String, Object> person = new HashMap<String, Object>(2);
person.put("name", "james");
person.put("age", 25);
if(!tdb.put("james", person)) {
System.out.println("put error:" + tdb.errmsg(tdb.ecode()));
}
Map<String, Object> ret = tdb.get("james");
for (String key : ret.keySet()) {
System.out.println(key + ":" + ret.get(key));
}
tdb.close();
}
}
3. Tokyo Tyrant (network interface of Tokyo Cabinet)
3.1 Tokyo Tyrant 란?
- Tokyo Cabinet을 클라이언트-서버 구조의 데이터베이스처럼 사용할 수 있게 해주는 인터페이스와 서버 프로세스 이다.
- client/server model, multi applications can access one database
- Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet.
- Though the DBM has high performance, you might bother in case that multiple processes share the same database, or remote processes access the database
- Tokyo Tyrant is provided for concurrent and remote connections to Tokyo Cabinet
- It is composed of the server process managing a database and its access library for client applications.
(클라이언트 응용 프로그램에 대한 데이터베이스와 ACCESS 라이브러리를 관리하는 서버 프로세스로 구성되어 있습니다.)
3.2 Tokyo Tyrant의 특징
- Linux/*BSD kernel의 epoll/[kqueue|http://en.wikipedia.org/wiki/Kqueue] mechanism과 thread-pool model을 구현하여 high concurrency를 제공한다.
- server와 clients가 TCP/IP(on)에 대한 simple binary protocol로 통신한다.
- memcached와 HTTP를 지원하며 거의 모든 플랫폼과 프로그램 언어를 Tokyo Tyrant에서 사용할 수 있도록 지원한다.
- Linux, FreeBSD, Mac OS X, Solaris에서만 사용가능 하다
- high availability
- hot backup and update log
- asynchronous replication between servers
- various database schema (using the abstract database API of Tokyo Cabinet)
- on-memory hash database API
- the on-memory tree database API
- the hash API
- the B+ tree database API
- the fixed-length database API
- the table database API
- pure script language interfaces
- Perl, Ruby, Java, Python, PHP, Erlang, etc...
3.3 주요기능
Asynchronous Replication
- Master-Slaves 구조와 Dual Master 구조 지원
Thread Pool Model
- hread Pool Model 지원 : 고정 개수의 Thread를 미리 생성하여 Pool을 구성하고 필요 할 때마다 Pool로 부터 할당
3.4 Tokyo Tyrant 설치 ( tokyotyrant-1.1.41.tar.gz )
wget http://fallabs.com/tokyotyrant/tokyotyrant-1.1.41.tar.gz
tar zxvf tokyotyrant-1.1.41.tar.gz
cd tokyotyrant-1.1.41
#tokyocabinet HOME 지정
./configure --prefix=/home/gurubee/cloud/env/tokyotyrant --with-tc=/home/gurubee/cloud/env/tokyocabinet
#================================================================
# Ready to make.
#================================================================
make
#================================================================
# Ready to install.
#================================================================
make install
#================================================================
# Thanks for using Tokyo Tyrant.
#================================================================
Tokyo Tyrant 실행
- ttserver 명령을 실행하여 Tokyo Tyrant 서버를 기동한다.
mkdir /home/gurubee/cloud/data/tc-data
/home/gurubee/cloud/env/tokyotyrant/bin/ttserver
-port 11211 -thnum 8 -dmn
-pid /home/gurubee/cloud/data/tc-data/ttserver.pid
-log /home/gurubee/cloud/data/tc-data/ttserver.log -le
-ulog /home/gurubee/cloud/data/tc-data/ -ulim 128m -sid 1
-rts /home/gurubee/cloud/data/tc-data/ttserver.rts /home/gurubee/cloud/data/tc-data/database.tcb
netstat -anp | grep LISTEN
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 19714/ttserver
- ttserver Option
- -host name : specify the host name or the address of the server. By default, every network address is bound.
- -port num : specify the port number. By default, it is 1978.
- -thnum num : specify the number of worker threads. By default, it is 8.
- -pid path : output the process ID into the file.
- -log path : output log messages into the file.
- -ulog path : specify the update log directory.
3.5 Tokyo Tyrant Example Code
C Example Code
#include <tcrdb.h>
#include <stdlib.h>
#include <stdbool.h>
#include <stdint.h>
int main(int argc, char **argv){
TCRDB *rdb;
int ecode;
char *value;
/* create the object */
rdb = tcrdbnew();
/* connect to the server */
if(!tcrdbopen(rdb, "localhost", 1978)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "open error: %s\n", tcrdberrmsg(ecode));
}
/* store records */
if(!tcrdbput2(rdb, "foo", "hop") ||
!tcrdbput2(rdb, "bar", "step") ||
!tcrdbput2(rdb, "baz", "jump")){
ecode = tcrdbecode(rdb);
fprintf(stderr, "put error: %s\n", tcrdberrmsg(ecode));
}
/* retrieve records */
value = tcrdbget2(rdb, "foo");
if(value){
printf("%s\n", value);
free(value);
} else {
ecode = tcrdbecode(rdb);
fprintf(stderr, "get error: %s\n", tcrdberrmsg(ecode));
}
/* close the connection */
if(!tcrdbclose(rdb)){
ecode = tcrdbecode(rdb);
fprintf(stderr, "close error: %s\n", tcrdberrmsg(ecode));
}
/* delete the object */
tcrdbdel(rdb);
return 0;
}
JAVA Example Code
import java.io.IOException;
import java.net.InetSocketAddress;
import tokyotyrant.RDB;
import tokyotyrant.transcoder.DoubleTranscoder;
import tokyotyrant.transcoder.IntegerTranscoder;
public class RDBExample {
public static void main(String[] args) throws IOException {
Object key;
Object value;
// create the object
RDB db = new RDB();
// connect to the server
db.open(new InetSocketAddress("localhost", 1978));
// store records
if (!db.put("foo", "hop")
|| !db.put("bar", "step")
|| !db.put("baz", "jump")) {
System.err.println("put error");
}
// retrieve records
value = db.get("foo");
if (value != null) {
System.out.println(value);
} else {
System.err.println("get error");
}
// traverse records
db.iterinit();
while ((key = db.iternext()) != null) {
value = db.get(key);
if (value != null) {
System.out.println(key + ":" + value);
}
}
// add int
db.put("int", 3, new IntegerTranscoder());
int i = db.addint("int", 4);
System.out.println(i);
// add double
db.put("d", 3.0D, new DoubleTranscoder());
double d = db.adddouble("d", 4.0D);
System.out.println(d);
// close the connection
db.close();
}
}
import tokyotyrant.MRDB;
import tokyotyrant.networking.NodeAddress;
import tokyotyrant.transcoder.DoubleTranscoder;
import tokyotyrant.transcoder.IntegerTranscoder;
public class MRDBExample {
public static void main(String[] args) throws Exception {
Object value;
// create the object
MRDB db = new MRDB();
// connect to the servers
db.open(NodeAddress.addresses("tcp://localhost:1978"));
// store records
if (!db.await(db.put("foo", "hop"))
|| !db.await(db.put("bar", "step"))
|| !db.await(db.put("baz", "jump"))) {
System.err.println("put error");
}
// retrieve records
value = db.await(db.get("foo"));
if (value != null) {
System.out.println(value);
} else {
System.err.println("get error");
}
// add int
db.put("int", 3, new IntegerTranscoder());
int i = db.await(db.addint("int", 4));
System.out.println(i);
// add double
db.put("d", 3.0D, new DoubleTranscoder());
double d = db.await(db.adddouble("d", 4.0D));
System.out.println(d);
// close the connections
db.close();
}
}
4. 참고자료
