LeoFS Storage Operation

LeoFS Storage’s Cluster Operation

Shell Description
leofs-adm detach <storage-node>
  • Remove the storage node from the LeoFS storage cluster
  • Current status: running | stop
leofs-adm suspend <storage-node>
  • Suspend a storage node for maintenance
  • This command is NOT similar to the detach command, just only to suspend the node.
  • While suspending, it rejects any requests
  • Current status: running
leofs-adm resume <storage-node>
  • Resume a storage node until a finished maintenance
  • Current status: suspended | restarted
leofs-adm start
  • Start LeoFS after distributing the RING from LeoFS Manager to LeoFS Storage and Gateway
leofs-adm rebalance
  • Commit detached and attached nodes to update the cluster and Ring(routing-table)
  • Rebalance objects in the cluster which is based on the updated cluster topology

detach <storage-node>

  • Remove the storage node from the storage cluster
$ leofs-adm detach [email protected]
OK
$ leofs-adm rebalance
OK

suspend <storage-node>

  • Suspend the storage node
  • While suspending, it rejects any requests
  • This command does NOT detach the node from the storage cluster
$ leofs-adm suspend [email protected]
OK

resume <storage-node>

  • Resume the storage node
$ leofs-adm resume [email protected]
OK

start

  • Start LeoFS after distributing the RING from LeoFS Manager to LeoFS Storage and LeoFS Gateway
$ leofs-adm start
OK

rebalance

  • Commit detached and attached node(s) to update the cluster and Ring(routing-table)
  • Rebalance objects in the cluster which is based on the updated cluster topology
$ leofs-adm rebalance
OK

Storage MQ Operation

Since

LeoFS v1.2.2

Overview

LeoFS Storage MQ is controllable mechanism manually. We’ve published mq-suspend and mq-resume command in leofs-adm script. In addition, LeoFS’s MQ mechanism is affected by the watchdog mechanism to reduce comsumption of message costs.

Description of the each MQ

Id Description
leo_delete_dir_queue After executed leofs-adm delete-bucket, messages of deletion object is added into the queue.
leo_comp_meta_with_dc_queue After executed leofs-adm recover-cluster, messages of comparison of metadata w/remote-node is added into the queue.
leo_sync_obj_with_dc_queue After executed leofs-adm recover-cluster, messages of synchronization of objects w/remote-node is added into the queue.
leo_recovery_node_queue After executed leofs-adm recover-node, messages of recovery objects of a node is added into the queue.
leo_async_deletion_queue After executed leofs-adm delete-bucket OR deletion of a bucket OR deletion of an object, message of async deletion of objs is added into the queue.
leo_rebalance_queue After executed leofs-adm rebalance, messages of rebalance is added in to the queue.
leo_sync_by_vnode_id_queue After executed leofs-adm rebalance, messages of synchronization of virtual-nodes is added into the queue.
leo_per_object_queue After executed leofs-adm rebalance OR leofs-adm recover-file OR leofs-adm recover-node OR fixing inconsistent object(s) with the recovery data mechanism, messages of recover inconsistent objects is added into the queue.

Commands

Shell Description
leofs-adm mq-stats <storage-node>
  • See the statuses of message queues used in LeoFS Storage
leofs-adm mq-suspend <storage-node> <mq-id>
  • Suspend a process consuming a message queue
  • Active message queues only can be suspended
  • While suspending, no messages are consumed
leofs-adm mq-resume <storage-node> <mq-id>
  • Resume a process consuming a message queue

mq-stats <storage-node>

You can check tatuses of the message queues in the LeoFS’s storage node. Explanation of columns:

Column Description
state A status of the MQ - [idling, running, suspending]
number of msgs A number of messages in the queue
batch of msgs A batch of messages of the MQ’s message-consumption
interval An interval time between the batch processing 
$ ./leofs-adm mq-stats [email protected]
              id                |    state    | number of msgs | batch of msgs  |    interval    |            description
--------------------------------+-------------+----------------|----------------|----------------|-----------------------------------
leo_delete_dir_queue            |   idling    | 0              | 1000           | 100            | delete directories
leo_comp_meta_with_dc_queue     |   idling    | 0              | 1000           | 100            | compare metadata w/remote-node
leo_sync_obj_with_dc_queue      |   idling    | 0              | 1000           | 100            | sync objs w/remote-node
leo_recovery_node_queue         |   idling    | 0              | 1000           | 100            | recovery objs of node
leo_async_deletion_queue        |   idling    | 0              | 1000           | 100            | async deletion of objs
leo_rebalance_queue             |   running   | 2167           | 1400           | 10             | rebalance objs
leo_sync_by_vnode_id_queue      |   idling    | 0              | 1000           | 100            | sync objs by vnode-id
leo_per_object_queue            |   idling    | 0              | 1000           | 100            | recover inconsistent objs

mq-suspend <storage-node> <mq-id>

Note

When turning on the watchdog mechanism, this command is ignored.

$ ./leofs-adm mq-suspend [email protected] leo_delete_dir_queue
OK

mq-resume <storage-node> <mq-id>

Note

When turning on the watchdog mechanism, this command is ignored.

$ ./leofs-adm mq-resume [email protected] leo_delete_dir_queue
OK