This document will capture the troubleshooting that the team performs regularly while tacking problems in our several environments. This could include tribal knowledge or take that a step further to include lessons learnedcaptures random notes. This might include troubleshooting, workarounds, or other information that not captured anywhere else.

Building the DevNet tool

Clone the dapixio DevNet repo located at dapixio/fio-devnet: devnet remote node management tool (github.com).

As this is a private repo, you will need to be granted access as well as have either a GitHub Api Token or other means of ‘cloning’ the repo.

Once the repo is cloned, cd to the fio-devnet/cmd/devnet

Execute the cmd, 'go build'

These steps were performed on a machine with all the necessary dependencies, i.e. go, c/c++ compiler.

Troubleshooting

Database Dirty Flag

Relevant Issues

1. V1 History config:

   1. https://developers.fioprotocol.io/docs/chain/node-build-history

   2. warning Without the history-index-state-db-size-mb and history-state-db-size-mb settings nodes may stop with the warning: Database has reached an unsafe level of usage, shutting down to avoid corrupting the database. Please increase the value set for chain-state-db-size-mb and restart the process!

2. Ye olde folder deletion problem where you have to manually delete some folders when doing a replay

   1. Jira Legacy
      server System JIRA serverId 5f0d8161-d4cf-3d17-96b1-53b2b2b5013d key BD-1953

   2. To solve this you have to delete the history and history-state folders if you are not running v1 history

BP Input

[Ross/EOSphere} Restart with a snapshot from here and check the nodeos usage example at the bottom.

...

Also delete the two history folders in the nodeos binary directory you started from.

Outcome

Thanks Ross; this particular setup has /var/lib/fio/ with data and history folders. It has no snapshots nor blocks dirs in data. There were two history* folders.

...