pruning parameter configuration of exchaind:
1. Default parameter --pruning default:
The state is saved every 10,000 blocks, and the state of the last 100 blocks is kept, and the pruning operation is performed every 10 blocks.
Equivalent to the following custom configuration:
--pruning=custom
--pruning-keep-recent=100
--pruning-keep-every=10000
--pruning-interval=10
--pruning-max-worldstate-num=200
2. Prune all snapshots --pruning everything:
All saved states will be deleted, only the current state will be stored, and the pruning operation will be performed every 10 blocks.
Pros: Minimal disk footprint, only the latest state is kept.
Equivalent to the following custom configuration:
--pruning=custom
--pruning-keep-recent=0
--pruning-keep-every=0
--pruning-interval=10
--pruning-max-worldstate-num=0
3. Keep all snapshots --pruning nothing:
All historical state will be preserved and nothing will be deleted, i.e. archive nodes.
Advantages: Store all block heights, never cut app.db, occupy the largest disk, and have the most complete data.
Equivalent to the following custom configuration:
--pruning=custom
--pruning-keep-recent=0
--pruning-keep-every=1
--pruning-interval=0
--pruning-max-worldstate-num=2^64-1
4. Customize --pruning custom
--pruning-keep-recent [x] # Keep snapshots of the latest x blocks, and blocks before x are automatically cut;
--pruning-keep-every [x] # keep a snapshot every x blocks;
--pruning-interval # pruning interval block;
--pruning-max-worldstate-num # Maximum number of saved states
1. Background
oec provides a variety of snapshots, including minimal snapshots, normal snapshots, and archive node snapshots. The characteristics and differences of different snapshots are detailed in here.
When oec is running, it also has a variety of pruning configuration parameters, allowing users to save block data according to actual conditions. Details of pruning configuration parameters reference.
Each configuration consumes disk differently. Among them, the archive node occupies the largest consumption, and the smallest snapshot consumes the least.
At present, according to the snapshot size, it is divided into four levels: s0, s1, s2, and s3:
- s0.
size is the smallest, only contains the state of the last block and this height.
The minimum data snapshot is the data cut to the last block, delete cs.wal, tx_index.db, watch.db data.
The advantages are: small amount of data, fast downloading of data, and rapid deployment of nodes;
The disadvantage is that historical data cannot be queried, including historical block data, transaction data, status data, event data, etc. The data before the block height cannot be queried, but after the synchronous block operation, the data after the block height can be queried.
- s1.
--pruning everything: larger than s0 size
Normal data snapshots have a medium amount of data, and only the state data of the latest height is kept. The previous state data has been pruned, and the block and states data have not been pruned. The block and state data include data from height 2322601 (mainnet) to current block height. You can query the current block data, transaction data, and status data, which can be used to continue to generate blocks after deployment.
- s2.
--pruning default: larger than s1 size
This data is the default configuration of the system and can be changed according to your needs.
- s3.
--pruning nothing: maximum size
Archived data snapshots have the largest amount of data, without any pruning. The block, state, and application data include all data from the height of 2322601 (main network) to the current block height, and can query historical block data, transaction data, status data, event data, etc.
The disadvantage is: the amount of data is very large, and it takes a certain amount of time to download the data.
2. Questions
How to minimize disk usage while meeting your own needs.
for example:
If you want data to occupy the least disk space and do not need to query historical data, then s0 is the most suitable, how to make it
If you want data to occupy less disk space, but at the same time retain historical block data, then s1 is the most suitable, how to make it
If you want to keep all the historical data and status, you can use a larger disk space, then s3 is the most suitable, how to make it
In addition, if the amount of data is large during the cutting process, what should I do if the execution fails?
How to compress only app data without cropping, how to compress only block data without cropping, etc.
3. Solutions
Prepare initial data
cd /data/pruning_node/${HOME}
cp -R ${ORIGIN_HOME}/exchaind/data .
cp -R ${ORIGIN_HOME}/exchaind/config . # only execute for the first time
Use the latest dev to compile exchaind
git clone -b dev https://github.com/okx/exchain.git
cd exchange
git pull # Make sure to be on the dev branch
make mainnet # Compile the mainnet, default WITH_ROCKSDB=true
make testnet # Compile the testnet, default WITH_ROCKSDB=true
Notice:
**make mainnet # When compiling the mainnet, be sure to use
make testnet # When compiling the testnet, be sure to use **
1. How to make s0
cd /data/pruning_node
exchaind data prune-compact all --home ${HOME}
- all: Crop and compress block and app data at the same time, process
application.db blockstore.db state.db.
After cropping, delete useless data to further reduce disk space usage
rm -r ${HOME}/data/tx_index.db
rm -r ${HOME}/data/watch.db
rm -r ${HOME}/data/cs.wal
rm -r ${HOME}/data/bloom.db
rm -r ${HOME}/data/evidence.db
2. How to make s1
cd /data/pruning_node
exchaind data prune-compact state --home ${HOME}
exchaind data prune-compact block -p=false --home ${HOME}
- state: crop and compress app data, process
application.db.
-p: Whether to perform the pruning operation, if false, only the compact operation will be performed.- block: crop and compress block data, process
blockstore.db state.db.
Note: block -p=false means not to crop but only to compress block data.
3. How to make s3
Since s3 contains the most complete data, it cannot be made through the command line, but it can be done by configuring --pruning nothing when starting the node.
Note: Archive data generation only works for newly generated blocks after configuring --pruning nothing, and does not work for historical data. If you need complete historical data, you can download the archive node snapshot
4. If the amount of data is too large, how to avoid OOM (memory overflow)
For example, the block data before 200w, 200w-300w, 300w-400w, 400w-500w can be cut segment by segment into s0 four times:
exchaind data prune-compact all -r 2000000 --home ${HOME}
exchaind data prune-compact all -r 3000000 --home ${HOME}
exchaind data prune-compact all -r 4000000 --home ${HOME}
exchaind data prune-compact all -r 5000000 --home ${HOME}
-r: Specifies the clipping height, the data before this height (excluding) will be cropped. If not specified, the last two height data are kept by default.
5. How to compress app data without cropping
exchaind data prune-compact state -p=false --home ${HOME}
exchaind data prune-compact state -p=false -r 100 --home xxx # -r specifies the clipping height, and the data before this height (excluding) will be clipped.
- state: pruning only compresses app data, processing
application.db.
-p: Whether to perform the pruning operation, if false, only the compact operation will be performed.
Note: state -p=false means no cropping and only app data compression.
6. How not to crop and only compress block data
exchaind data prune-compact block -p=false --home ${HOME}
exchaind data prune-compact block -p=false -r 100 --home xxx # -r specifies the clipping height, and the data before this height (excluding) will be clipped.
- block: pruning only compresses block data, processing
blockstore.db state.db.
-p: Whether to perform the pruning operation, if false, only the compact operation will be performed.
Note: block -p=false means not to crop but only to compress block data.
How to configure the startup parameter pruning
The following two configurations are recommended:
rpc node, configured to prune all snapshots --pruning everything
All saved states will be deleted, only the current state will be stored, and the pruning operation will be performed every 10 blocks.
Pros: Minimal disk footprint, only the latest state is kept.
Archive node, keep all snapshots --pruning nothing
All historical state will be preserved and nothing will be deleted, i.e. archive nodes.
Query DB information
exchaind data query block --home xxx
exchaind data query state --home xxx
- block: Query the block height contained in
blockstore.db.
- state: Query the state version contained in
application.db.
--home: Specify the data directory, same as exchaind start --home.