Configuration

Page for release 1.8.1. See this page for: release 1.8.2, release 1.8.4

Namespace configuration

Chunk size

Uploaded content are split into chunks, whose size cannot exceed the chunk size. Chunk size is configured in @conscience as follow:

[Plugin.conscience]
param_namespace=$namespace
param_chunk_size=$your_value

Value is in byte.<br> Current chunk size is displayed by gridcluster command

[root@singlenode ~]# gridcluster TESTNS

NAMESPACE INFORMATION

                Name : TESTNS
          Chunk size : 10485760 bytes
…

Quota

A quota can be applied at the namespace level, aggregating each container size at once. Quota is configured in @conscience as follow:

[Plugin.conscience]
param_namespace=$namespace
param_option.quota_$namespace=$your_value

Value is in byte.<br> Current quota is displayed by gridcluster command

[root@singlenode ~]# gridcluster TESTNS

NAMESPACE INFORMATION

                Name : TESTNS
          Chunk size : 10485760 bytes
              Option : quota_TESTNS = 1000000

Virtual Namespace

Virtual namespace (VNS) can be created at will ‘under’ a physical namespace. It is possible to create containers with the same name in two different VNS. VNS are declared in @conscience as follow:

[Plugin.conscience]
param_namespace=$namespace
param_option.vns_list=$namespace.$virtual_namespace1, $namespace.$virtual_namespace2

VNS syntax is: $namespace “.” $virtualnamespace. The ‘master’ namespace must appear, followed by “.” dot char, followed by the actual virtual namespace
VNS list is comma separated

Example for namespace TESTNS :

param_option.vns_list=TESTNS.Virtual1, TESTNS.Virtual2

VNS are displayed by gridcluster command

[root@singlenode ~]# gridcluster TESTNS

NAMESPACE INFORMATION

                Name : TESTNS
          Chunk size : 10485760 bytes
              Option : writable_vns = TESTNS.Virtual2,TESTNS.Virtual1,TESTNS
              Option : vns_list = TESTNS.Virtual1,TESTNS.Virtual2

WORM

WORM mode prevents content from being altered. It means that PUT, APPEND and DELETE operations are forbidden on existing content.
WORM mode is configured in @conscience as follow:

[Plugin.conscience]
path=/usr/local/lib64/grid/msg_conscience.so
param_namespace=$namespace
param_option.worm=on

Mode can be on or off.
WORM mode is displayed by gridcluster command

[root@singlenode conf]# gridcluster TESTNS

NAMESPACE INFORMATION

                Name : TESTNS
          Chunk size : 10485760 bytes
              Option : worm = on

Storage policy

A storage policy is triplet of a storage class, a data security level and a data treatment:

  • The storage class defines the type of storage, whether it is local or attached, fast or big.
  • The data security level defines the storage algorithm to be used, duplication, parity or none
  • The data treatment defines the treatment to be applied to chunk before being physically stored on disk: compression, cypher or none.

Current implementation is limited to data security level duplication or none.

Storage policy is defined in the conscience for each namespace. The administrator must define the default policy, and a configuration file where every policy is defined, as follow:

param_option.storage_policy=TWOCOPIES
param_storage_conf=/usr/local/share/grid/storage_conf

In this example, the default policy is named TWOCOPIES, which is self-explanatory. It will duplicate each chunk only once, as described in the configuration file

[STORAGE_POLICY]
TWOCOPIES=DUMMY:DUPONETWO:NONE
FIVECOPIES=DUMMY:DUPONEFIVE:NONE
NONE=DUMMY:NONE:NONE

[DATA_SECURITY]
DUPONETWO=DUP:distance=1|nb_copy=2
DUPONEFIVE=DUP:distance=1|nb_copy=5
NONE=DUP:0:1

[DATA_TREATMENTS]

Score

The notion of score in Redcurrant is an indicator of service activity, calculated as an integer value in the range [0..100].

  • 100 means full availability of the service
  • 0 means total unavailability of the service

Conscience gathers statistics from each service through gridagent processes located on each Redcurrant node. Then, for each type of service, conscience computes a score given a specific formula.

A default rule in conscience set a score to 100 as follow:

param_service.default.score_timeout=300
param_service.default.score_variation_bound=5
param_service.default.score_expr=100

Custom formula can be used; here is a square root for meta1:

param_service.meta1.score_timeout=300
param_service.meta1.score_variation_bound=5
param_service.meta1.score_expr=root(2,((num stat.cpu) * (num stat.req_idle)))

Here is a cubic root for meta2:

param_service.meta2.score_timeout=300
param_service.meta2.score_variation_bound=5
param_service.meta2.score_expr=((num stat.space)>=5) * root(3,((num stat.cpu)*(num stat.req_idle)*(num stat.space)))

Each service provides its own statistics, which can be used in formula. In a running redcurrant, list of statistics can be found with gridcluster command. Each “stat.*” key is a valid metric in a formula.

[root@singlenode conf]# gridcluster -l
TESTNS|meta2|192.168.244.143:6006|score=95|tag.db_config=sqlite:///DATA/TESTNS/singlenode/meta2-2?auto_commit=1&use_hash=true&hash_depth=2|stat.cpu=97|stat.req_idle=100|stat.space=90|stat.io=100|tag.up=true
TESTNS|meta1|192.168.244.143:6002|score=10|tag.vol=/DATA/TESTNS/singlenode/meta1-1|stat.req_idle=1.234000|stat.cpu=98|stat.space=90|stat.io=100|tag.up=true
TESTNS|meta1|192.168.244.143:6003|score=10|tag.vol=/DATA/TESTNS/singlenode/meta1-2|stat.req_idle=1.234000|stat.cpu=98|stat.space=90|stat.io=100|tag.up=true
TESTNS|meta1|192.168.244.143:6004|score=10|tag.vol=/DATA/TESTNS/singlenode/meta1-3|stat.req_idle=1.234000|stat.cpu=98|stat.space=90|stat.io=100|tag.up=true
TESTNS|meta2|192.168.244.143:6005|score=95|tag.db_config=sqlite:///DATA/TESTNS/singlenode/meta2-1?auto_commit=1&use_hash=true&hash_depth=2|stat.cpu=97|stat.req_idle=100|stat.space=90|stat.io=100|tag.up=true

score_timeout (in second): if no metric is received from this service, score is automatically set to 0
score_variation_bound: maximum increment of score for a service. This avoid score flip, from 0 to 100 and vice versa.

Container configuration

Quota

Container’s quota is a global directive applied at conscience level. Quota is applied regarding the actual container size, not including the uploaded content size.

Example:
Container quota is 100MB
Current container size is 80MB
A client can put a content of 30MB once, and then container size will reach 110MB. No more PUT/APPEND operation will be allowed afterwards, until size is reduced.

Container’s quota is configured in @conscience as follow:

[Plugin.conscience]
param_namespace=$namespace
param_chunk_size=$your_value
param_option.container_max_size=$your_quota

Container’s quota is displayed by gridcluster command

[root@singlenode conf]# gridcluster TESTNS

NAMESPACE INFORMATION

                Name : TESTNS
          Chunk size : 10485760 bytes
              Option : container_max_size = 8192

NOTE: container quota is applied to a namespace and EACH NESTED VIRTUAL NAMESPACE

Versioning

Redcurrant has the ability to store versioned contents in its directory. This means that contents can be uploaded several times with the same name in the same container, and each upload creates a new version.
A container can be configured as versioned or not versioned (default behavior before 1.8). This can be specified when the container is created and a default versioning policy can be configured at the namespace level.

The versioning policy is defined as an integer with the following meaning:

  • 0 : versioning is off.
  • -1 : versioning is on with unlimited number of versions.
  • N : versioning is on with a limit of N versions per content.

To configure the default versioning policy at the namespace level, change the conscience configuration as follow:

[Plugin.conscience]
param_namespace=$namespace
param_chunk_size=$your_value
param_option.meta2_max_versions=10	

User Tools