Updated readme
General description What storage to choose
This commit is contained in:
parent
f1e78c8f1b
commit
ccb2e46400
122
README.md
122
README.md
|
|
@ -1,23 +1,46 @@
|
|||
# Burlesque
|
||||
|
||||
Burlesque is a [message processing queue](http://en.wikipedia.org/wiki/Message_queue) writen in Go. It exposes queues using the HTTP API that allows publishing messages and subscribing to them. See the [API](#api) section for more details.
|
||||
Burlesque is a [message processing queue](http://en.wikipedia.org/wiki/Message_queue) writen in [Go](http://golang.org/). It exposes queues using the [pub/sub HTTP API](#api).
|
||||
|
||||
Burlesque stores messages inside a [Kyoto Cabinet](http://fallabs.com/kyotocabinet/) database.
|
||||
The general purpose of this queue is to provide tool for inter-process comutication with a memory efficient persisted storage for messages (usually a delayed job description serialized in JSON) published by the application server and later retrieved by other application workers.
|
||||
|
||||
Subscription is done using [long polling](http://en.wikipedia.org/wiki/Push_technology#Long_polling) technique. When application worker subscribes to a queue which is empty at the moment, connection is kept open until another client publishes a message to this queue, or the first client disconnects. If there is a message in the queue it will be removed from the queue and returned to the client.
|
||||
|
||||
Burlesque uses [Kyoto Cabinet](http://fallabs.com/kyotocabinet/) to store messages, which is a powerfull DIY database. Usage of Kyoto Cabinet is thoroughly described in the [storage](#storage) section of this document.
|
||||
|
||||
#### Contents
|
||||
|
||||
* [Installation](#installation)
|
||||
* [Building on OSX](#building-on-osx)
|
||||
* [Starting](#starting)
|
||||
* [Storage](#storage)
|
||||
* [In-memory databases](#in-memory-databases)
|
||||
* [Persistent databases](#persistent-databases)
|
||||
* [Tuning parameters](#tuning-parameters)
|
||||
* [Support of tuning parameters by databases](#support-of-tuning-parameters-by-databases)
|
||||
* [What storage to choose](#what-storage-to-choose)
|
||||
* [Production](#production)
|
||||
* [Development](#development)
|
||||
* [API](#api)
|
||||
* [Publish](#publish)
|
||||
* [Subscribe](#subscribe)
|
||||
* [Status](#status)
|
||||
* [Debug](#debug)
|
||||
|
||||
## Installation
|
||||
|
||||
OSX:
|
||||
Download and extract the [latest release](https://github.com/KosyanMedia/burlesque/releases). That's it.
|
||||
|
||||
### Building on OSX
|
||||
|
||||
First install [Homebrew](http://brew.sh/). Using Homebrew install Go language compiler and tools. Then install Kyoto Cabinet library.
|
||||
|
||||
```
|
||||
brew install go
|
||||
brew install kyoto-cabinet
|
||||
go get github.com/KosyanMedia/burlesque
|
||||
```
|
||||
|
||||
Linux:
|
||||
```
|
||||
...
|
||||
```
|
||||
|
||||
## Starting
|
||||
|
||||
Use the following arguments to the `burlesque` executable:
|
||||
|
|
@ -29,11 +52,23 @@ Use the following arguments to the `burlesque` executable:
|
|||
| `-port` | Server HTTP port | `4401` |
|
||||
| `-rollbar` | [Rollbar](https://rollbar.com/) token | ||
|
||||
|
||||
## Storage argument
|
||||
#### Example
|
||||
|
||||
```bash
|
||||
wget -O burlesque.zip https://github.com/KosyanMedia/burlesque/archive/1.0.0.zip
|
||||
unzip burlesque.zip
|
||||
./burlesque
|
||||
```
|
||||
|
||||
By default Burlesque starts on port `4401` in development mode and uses in-memory database `ProtoHashDB`.
|
||||
|
||||
## Storage
|
||||
|
||||
`-storage` argument defines a way the data will be stored into a database. You can read more on Kyoto Cabinet database types [here](http://fallabs.com/kyotocabinet/spex.html#tutorial_dbchart).
|
||||
|
||||
If you need a **temporary in-memory** storage use the following symbols as the `-storage` value:
|
||||
### In-memory databases
|
||||
|
||||
If you need a temporary in-memory storage use the following symbols as the `-storage` value:
|
||||
|
||||
| Value | Database Type |
|
||||
| ----- | ------------- |
|
||||
|
|
@ -43,9 +78,11 @@ If you need a **temporary in-memory** storage use the following symbols as the `
|
|||
| `*` | `CacheDB` Cache hash database. On-memory database featuring [LRU](http://en.wikipedia.org/wiki/Cache_algorithms#Examples) deletion |
|
||||
| `%` | `GrassDB` Cache tree database. On-memory database of B+ tree: cache with order |
|
||||
|
||||
In order to use a **persistent database** use the path to the database file (or directory) as the `-storage` argument value. File extension in the database path defines the type of the database created.
|
||||
#### Example: `-`
|
||||
|
||||
**Example:** `/path/to/my/storage.kch`
|
||||
### Persistent databases
|
||||
|
||||
In order to use a persistent database use the path to the database file (or directory) as the `-storage` argument value. File extension in the database path defines the type of the database created.
|
||||
|
||||
| File Extension | Database Type |
|
||||
| -------------- | ------------- |
|
||||
|
|
@ -55,9 +92,11 @@ In order to use a **persistent database** use the path to the database file (or
|
|||
| `kcf` | `ForestDB` Directory tree database. Directory database of B+ tree: huge DBM with order |
|
||||
| `kcx` | `TextDB` Plain text database. Emulation to handle a plain text file as a database |
|
||||
|
||||
In addition to defining database type you can also add [tuning parameters](http://fallabs.com/kyotocabinet/spex.html#tips) to the `-storage` argument. Tuning parameters are separated by the `#` symbol, parameters' name and value are separated by the `=` symbol.
|
||||
#### Example: `/path/to/my/storage.kch`
|
||||
|
||||
**Example:** `storage.kch#opts=c#zcomp=gz#msiz=524288000`
|
||||
### Tuning parameters
|
||||
|
||||
In addition to defining database type you can also add [tuning parameters](http://fallabs.com/kyotocabinet/spex.html#tips) to the `-storage` argument. Tuning parameters are separated by the `#` symbol, parameters' name and value are separated by the `=` symbol.
|
||||
|
||||
The table below describes tuning parameters.
|
||||
|
||||
|
|
@ -65,8 +104,8 @@ The table below describes tuning parameters.
|
|||
| ---------- | ----------- |
|
||||
| `apow` | Power of the alignment of record size |
|
||||
| `bnum` | Base hash table size (number of buckets of the hash table) |
|
||||
| `capcnt` | Capacity limit by the number of records (`capcnt=10000` means "keep in memory 10,000 records maximum) |
|
||||
| `capsiz` | Capacity limit by the size of records (`capsiz=536870912` means "keep in memory all the records that fit into 512 megabytes) |
|
||||
| `capcnt` | Capacity limit by the number of records (`#capcnt=10000` means "keep in memory 10,000 records maximum) |
|
||||
| `capsiz` | Capacity limit by the size of records (`#capsiz=536870912` means "keep in memory all the records that fit into 512 megabytes) |
|
||||
| `dfunit` | Unit step number of auto defragmentation |
|
||||
| `fpow` | Power of the capacity of the free block pool |
|
||||
| `log` | Path to the log file. Use `-` for the STDOUT, or `+` for the STDERR |
|
||||
|
|
@ -80,6 +119,9 @@ The table below describes tuning parameters.
|
|||
| `zcomp` | Compression library: `zlib` for the [ZLIB raw](http://en.wikipedia.org/wiki/Zlib#Encapsulation) compressor, `def` for the ZLIB [deflate](http://en.wikipedia.org/wiki/DEFLATE) compressor, `gz` for the ZLIB [gzip](http://en.wikipedia.org/wiki/Gzip) compressor, `lzo` for the [LZO](http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Oberhumer) compressor, `lzma` for the [LZMA](http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm) compressor, or `arc` for the [Arcfour](http://en.wikipedia.org/wiki/RC4) cipher |
|
||||
| `zkey` | Cipher keyword used with compression |
|
||||
|
||||
#### Example: `storage.kch#opts=c#zcomp=gz#msiz=524288000`
|
||||
|
||||
### Support of tuning parameters by databases
|
||||
|
||||
The table below describes support of these parameters by the **in-memory** database types.
|
||||
|
||||
|
|
@ -104,8 +146,6 @@ The table below describes support of these parameters by the **persistent** data
|
|||
| ---------- | :------: | :------: | :-----: | :--------: | :------: |
|
||||
| `apow` | **Yes** | **Yes** | — | — | — |
|
||||
| `bnum` | **Yes** | **Yes** | — | — | — |
|
||||
| `capcnt` | — | — | — | — | — |
|
||||
| `capsiz` | — | — | — | — | — |
|
||||
| `dfunit` | **Yes** | **Yes** | — | — | — |
|
||||
| `fpow` | **Yes** | **Yes** | — | — | — |
|
||||
| `log` | **Yes** | **Yes** | **Yes** | **Yes** | **Yes** |
|
||||
|
|
@ -119,9 +159,31 @@ The table below describes support of these parameters by the **persistent** data
|
|||
| `zcomp` | **Yes** | **Yes** | **Yes** | **Yes** | — |
|
||||
| `zkey` | **Yes** | **Yes** | **Yes** | **Yes** | — |
|
||||
|
||||
## What storage to choose
|
||||
|
||||
### Production
|
||||
|
||||
For production usage it is strongly recommended to choose a **persistent** database. Internally Burlesque uses Kyoto Cabinet as a persisted hash-table, so using `HashDB` would be a smart choise.
|
||||
|
||||
If the average message size expected to be more than 1KB then compression should be considered as an option. To enable compression you need to pass `opts` tuning parameter to the database path with value `c` (`#opts=c`), you also need to define compression algorithm using the `zcomp` parameter (e.g `#zcomp=gz`).
|
||||
|
||||
You can define maximum memory limit; when the limit is reached new records are swapped to disk. Memory limit is defined by passing `msiz` parameter with value in bytes (e.g `#msiz=524288000`)
|
||||
|
||||
So, to use a persisted hash database with enabled compression and 512MB memory limit the `-storage` argument value is `storage.kch#opts=c#zcomp=gz#msiz=524288000`.
|
||||
|
||||
#### Further tuning
|
||||
|
||||
If queues are kept empty all at relatively small size, `bnum` option might be considered (e.g `#bnum=1000`)
|
||||
|
||||
### Development
|
||||
|
||||
If development database don't need to be persisted consider using `ProtoHashDB` (which locks the whole table), `StashDB` (locks record) or `CacheDB` (locks record using a mutex). By default `ProtoHashDB` is used.
|
||||
|
||||
## API
|
||||
|
||||
## `/publish`
|
||||
All endpoints exposed by the API are described below.
|
||||
|
||||
## Publish
|
||||
|
||||
Publishes a message to the given queue. If there is a connection waiting to recieve a message from this queue, the message would be transfered directly to the awaiting connection.
|
||||
|
||||
|
|
@ -129,39 +191,39 @@ Publication can be done via both `GET` and `POST` methods. To publish a message
|
|||
|
||||
Server will respond with `OK` message.
|
||||
|
||||
**Example:**
|
||||
#### Example
|
||||
```
|
||||
/publish?queue=urgent&msg=Process+this+message+as+soon+as+possible!
|
||||
```
|
||||
Response:
|
||||
Response
|
||||
```
|
||||
OK
|
||||
```
|
||||
|
||||
## `/subscribe`
|
||||
## Subscribe
|
||||
|
||||
Tries to fetch a message from one of the queues given. If there is a message at least in one of these queues, the message will be removed from the queue and returned as response body. The name of the queue from which the message was taken from will be provided inside a `Queue` response header.
|
||||
|
||||
Subscription is always done via `GET` method. To fetch a message from a queue use the name of the queue as the `queues` argument value. Multiple queue names could be passed separated with the `,` (quote) character.
|
||||
|
||||
**Example:**
|
||||
#### Example
|
||||
```
|
||||
/subscribe?queues=urgent,someday
|
||||
```
|
||||
Response:
|
||||
Response
|
||||
```
|
||||
Process this message as soon as possible!
|
||||
```
|
||||
|
||||
## `/status`
|
||||
## Status
|
||||
|
||||
Displays information about the queues, their messages and current subscriptions encoded in JSON format.
|
||||
|
||||
**Example:**
|
||||
#### Example
|
||||
```
|
||||
/status
|
||||
```
|
||||
Response:
|
||||
Response
|
||||
```json
|
||||
{
|
||||
"urgent": {
|
||||
|
|
@ -175,15 +237,15 @@ Response:
|
|||
}
|
||||
```
|
||||
|
||||
## `/debug`
|
||||
## Debug
|
||||
|
||||
Displays debug information about the queue process. Currenty displays the number of goroutines only.
|
||||
|
||||
**Example:**
|
||||
#### Example
|
||||
```
|
||||
/debug
|
||||
```
|
||||
Response:
|
||||
Response
|
||||
```json
|
||||
{
|
||||
"goroutines": 13
|
||||
|
|
|
|||
Loading…
Reference in New Issue