Gregory Eremin 707a61d134 | ||
---|---|---|
.gitignore | ||
README.md | ||
config.go | ||
counter.go | ||
logging.go | ||
main.go | ||
queue.go | ||
request.go | ||
server.go | ||
state.go | ||
storage.go |
README.md
Burlesque
Burlesque is a message processing queue writen in Go. It exposes queues using the pub/sub HTTP API.
The general purpose of this queue is to provide tool for inter-process comutication with a memory efficient persisted storage for messages (usually a delayed job description serialized in JSON) published by the application server and later retrieved by other application workers.
Subscription is done using long polling technique. When application worker subscribes to a queue which is empty at the moment, connection is kept open until another client publishes a message to this queue, or the first client disconnects. If there is a message in the queue it will be removed from the queue and returned to the client.
Burlesque uses Kyoto Cabinet to store messages, which is a powerfull DIY database. Usage of Kyoto Cabinet is thoroughly described in the storage section of this document.
Contents
Installation
Download and extract the latest release. That's it.
Building on OSX
First install Homebrew. Using Homebrew install Go language compiler and tools. Then install Kyoto Cabinet library.
brew install go
brew install kyoto-cabinet
go get github.com/KosyanMedia/burlesque
Starting
Use the following arguments to the burlesque
executable:
Argument | Description | Defaults |
---|---|---|
-storage |
Kyoto Cabinet storage path (e.g. storage.kch#msiz=524288000 ) |
- |
-environment |
Process environment: development or production |
development |
-port |
Server HTTP port | 4401 |
-rollbar |
Rollbar token |
Example
wget -O burlesque.zip https://github.com/KosyanMedia/burlesque/archive/1.0.0.zip
unzip burlesque.zip
./burlesque
By default Burlesque starts on port 4401
in development mode and uses in-memory database ProtoHashDB
.
Storage
-storage
argument defines a way the data will be stored into a database. You can read more on Kyoto Cabinet database types here.
In-memory databases
If you need a temporary in-memory storage use the following symbols as the -storage
value:
Value | Database Type |
---|---|
- |
ProtoHashDB Prototype hash database. On-memory database implemented with std::unorderd_map |
+ |
ProtoTreeDB Prototype tree database. On-memory database implemented with std::map |
: |
StashDB Stash database. On-memory database saving memory |
* |
CacheDB Cache hash database. On-memory database featuring LRU deletion |
% |
GrassDB Cache tree database. On-memory database of B+ tree: cache with order |
Example: -
Persistent databases
In order to use a persistent database use the path to the database file (or directory) as the -storage
argument value. File extension in the database path defines the type of the database created.
File Extension | Database Type |
---|---|
kch |
HashDB File hash database. File database of hash table: typical DBM |
kct |
TreeDB File tree database. File database of B+ tree: DBM with order |
kcd |
DirDB Directory hash database. Respective files in a directory of the file system |
kcf |
ForestDB Directory tree database. Directory database of B+ tree: huge DBM with order |
kcx |
TextDB Plain text database. Emulation to handle a plain text file as a database |
Example: /path/to/my/storage.kch
Tuning parameters
In addition to defining database type you can also add tuning parameters to the -storage
argument. Tuning parameters are separated by the #
symbol, parameters' name and value are separated by the =
symbol.
The table below describes tuning parameters.
Parameter | Description |
---|---|
apow |
Power of the alignment of record size |
bnum |
Base hash table size (number of buckets of the hash table) |
capcnt |
Capacity limit by the number of records (#capcnt=10000 means "keep in memory 10,000 records maximum) |
capsiz |
Capacity limit by the size of records (#capsiz=536870912 means "keep in memory all the records that fit into 512 megabytes) |
dfunit |
Unit step number of auto defragmentation |
fpow |
Power of the capacity of the free block pool |
log |
Path to the log file. Use - for the STDOUT, or + for the STDERR |
logkinds |
Kinds of logged messages. The value can be debug , info , warn or error |
logpx |
Prefix of each log message |
msiz |
Expected database memory usage |
opts |
Additional options: s , l and c (can be specified together, e.g lc ). s is for "small" and reduces the width of record addressing from 6 bytes to 4 bytes. As the result, the footprint for each record is reduced from 16 bytes to 12 bytes. However, it limits the maximum size of the database file up to 16GB. l is for "linear" and changes the data structure of the collision chain of hash table from binary tree to linear linked list. c enables compression of the record values. If the value is bigger than 1KB compression is effective. |
pccap |
Capacity size of the page cache |
psiz |
Page size |
rcomp |
Comparator used to compare key names. lex for the lexical comparator, dec for the decimal comparator, lexdesc for the lexical descending comparator, or decdesc for the decimal descending comparator |
zcomp |
Compression library: zlib for the ZLIB raw compressor, def for the ZLIB deflate compressor, gz for the ZLIB gzip compressor, lzo for the LZO compressor, lzma for the LZMA compressor, or arc for the Arcfour cipher |
zkey |
Cipher keyword used with compression |
Example: storage.kch#opts=c#zcomp=gz#msiz=524288000
Support of tuning parameters by databases
The table below describes support of these parameters by the in-memory database types.
Parameter | ProtoHashDB |
ProtoTreeDB |
StashDB |
CacheDB |
GrassDB |
---|---|---|---|---|---|
bnum |
— | — | Yes | Yes | Yes |
capcnt |
— | — | — | Yes | — |
capsiz |
— | — | — | Yes | — |
log |
Yes | Yes | Yes | Yes | Yes |
logkinds |
Yes | Yes | Yes | Yes | Yes |
logpx |
Yes | Yes | Yes | Yes | Yes |
opts |
— | — | — | Yes | Yes |
pccap |
— | — | — | — | Yes |
psiz |
— | — | — | — | Yes |
rcomp |
— | — | — | — | Yes |
zcomp |
— | — | — | Yes | Yes |
zkey |
— | — | — | Yes | Yes |
The table below describes support of these parameters by the persistent database types.
Parameter | HashDB |
TreeDB |
DirDB |
ForestDB |
TextDB |
---|---|---|---|---|---|
apow |
Yes | Yes | — | — | — |
bnum |
Yes | Yes | — | — | — |
dfunit |
Yes | Yes | — | — | — |
fpow |
Yes | Yes | — | — | — |
log |
Yes | Yes | Yes | Yes | Yes |
logkinds |
Yes | Yes | Yes | Yes | Yes |
logpx |
Yes | Yes | Yes | Yes | Yes |
msiz |
Yes | Yes | — | — | — |
opts |
Yes | Yes | Yes | Yes | — |
pccap |
— | Yes | — | Yes | — |
psiz |
— | Yes | — | Yes | — |
rcomp |
— | Yes | — | Yes | — |
zcomp |
Yes | Yes | Yes | Yes | — |
zkey |
Yes | Yes | Yes | Yes | — |
What storage to choose
Production
For production usage it is strongly recommended to choose a persistent database. Internally Burlesque uses Kyoto Cabinet as a persisted hash-table, so using HashDB
would be a smart choise.
If the average message size expected to be more than 1KB then compression should be considered as an option. To enable compression you need to pass opts
tuning parameter to the database path with value c
(#opts=c
), you also need to define compression algorithm using the zcomp
parameter (e.g #zcomp=gz
).
You can define maximum memory limit; when the limit is reached new records are swapped to disk. Memory limit is defined by passing msiz
parameter with value in bytes (e.g #msiz=524288000
)
So, to use a persisted hash database with enabled compression and 512MB memory limit the -storage
argument value is storage.kch#opts=c#zcomp=gz#msiz=524288000
.
Further tuning
If queues are kept empty all at relatively small size, bnum
option might be considered (e.g #bnum=1000
)
Development
If development database don't need to be persisted consider using ProtoHashDB
(which locks the whole table), StashDB
(locks record) or CacheDB
(locks record using a mutex). By default ProtoHashDB
is used.
API
All endpoints exposed by the API are described below.
Publish
Publishes a message to the given queue. If there is a connection waiting to recieve a message from this queue, the message would be transfered directly to the awaiting connection.
Publication can be done via both GET
and POST
methods. To publish a message via GET
method use the queue
argument to pass queue name and the msg
argument to pass message body. To publish a message via POST
method pass message body via request body instead of the msg
argument.
Server will respond with OK
message.
Example
curl '127.0.0.1:4401/publish?queue=urgent' -d 'Process this message as soon as possible!'
Response
OK
Subscribe
Tries to fetch a message from one of the queues given. If there is a message at least in one of these queues, the message will be removed from the queue and returned as response body. The name of the queue from which the message was taken from will be provided inside a Queue
response header.
Subscription is always done via GET
method. To fetch a message from a queue use the name of the queue as the queues
argument value. Multiple queue names could be passed separated with the ,
(quote) character.
Example
curl '127.0.0.1:4401/subscribe?queues=urgent,someday'
Response
Process this message as soon as possible!
Status
Displays information about the queues, their messages and current subscriptions encoded in JSON format.
Example
curl 127.0.0.1:4401/status
Response
{
"urgent": {
"messages": 0,
"subscriptions": 0
},
"someday": {
"messages": 0,
"subscriptions": 0
}
}
Debug
Displays debug information about the queue process. Currenty displays the number of goroutines only.
Example
curl 127.0.0.1:4401/debug
Response
{
"goroutines": 13
}