FIO Off Chain State

Owned by Ed Rotthoff
Last updated: Feb 09, 2022
5 min read

Goal

The FIO protocol would like feedback from block producers regarding data management technologies, data management approaches/tactics that should be considered as the FIO protocol goes forward in designing and implementing a layer 2 solution.

We also want your support in the identification and enablement of collaboration paths with key thought leaders that may be helpful in identifying in contributing to design and development of this FIO layer 2 solution. These collaborators may be advisory, or compensated rolls on this project.

Overview

This document discusses the benefits and advantages of providing the FIO protocol with an off-chain state database. This database is envisioned as containing both FIO protocol state information, as well as supplemental information that enriches the FIO user experience.

 

Staffing and Duration Considerations

This project is complex and may require dedicated core engineering team focus.

to adequately identify resources it will be best to establish the scope of the project and perform a staffing analysis after the scope is agreed. These positions listed are based upon assumptions of max scope.

tech oversight of the FIO lead architect. (50% of available bandwidth)

this project will require a database architect, this is a full time roll 180k base and 40k incentive per year.

this project may also require a distributed database administrator/egineer. 120k base and 40k incentive per year.

 

Background

Ledger technology implemented on the EOSIO platform has limited query-ability. The state model which is represented in the chains state table structure is not query-able using SQL like queries. A secondary concern is that the state model represented on chain cannot be supplemented with additional information targeted to enrich the user experience. The maintainability of the chain, and the performance of the chain both can become encumbered when state becomes too large, or too complex/varied in its structure. Transaction processing (and thus block production) become encumbered as the size and complexity of state grows.

EOSIO Tables are accessible using models of interaction which can be summarized as follows:

Table Query – The EOSIO protocol provides a get table call which can be used to query singular tables based on parameters that allow paging through results using the indexes that have been implemented into each table.

Chain plugin – The EOSIO protocol core code has a plug in model that provides a chain plugin. This plugin provides a set of programmed end points that may contain business logic and return structured results to callers.

Both of these mechanisms of query have restrictions on time execution, and so are limited as to the amount of results they may return during an invocation.

EOSIO also provides a history plugin which can be programmed to do various actions to manage data and provide notifications of various events on chain.

When we consider solutions to these limitations stated above, we naturally think “Well why not put the contents of state into an offline relational database somehow using a history node and do whatever is necessary”.

When we consider placing data into offline storage, we must consider that we must be able to verify that the offline data matches what is in state on the chain at any time.

There are established patterns of doing this off chain reflection which have been in practice since before EOSIO.

There are tools which have been developed for this purpose. History plugins, DEMUX technology, and many others.

We desire to explore and document the possible options FIO might use in collboration with FIO block producers, and with the larger EOSIO and crypto community. Our goal is the discovery of the best path for the FIO protocol going forward.

Key Considerations

When trying to solve this off chain state management for the FIO protocol, we would like to consider some of the following issues and requirements.

  1. We desire ease of readability for FIO state and supplemental information.

  2. We desire ease of deployability and increased manageability.

    1. For BPs.

      1. ease of playback.

      2. ease of data capture.

      3. ease of management of operational data.

    2. For integrators, we want to free them of the burden of having to upgrade nodes every time new feature function is added to the FIO protocol.

  3. We desire to increase FIO protocol performance.

    1. decrease size and complexity of all on chain state data.

      1. increase query performance.

      2. increase transaction performance when making blocks

      3. increase max users capability

      4. increase max volume capability.

      5. there are other perf advantages we can list that come from decreasing/optimizing state size.

  4. We desire to add supplemental information to whatever data model we provide for off chain state.

    1. We want to provide a state model that allows on chain state size to be reduced significantly.

  5. We want to provide a scalable and robust solution for populating off chain state.

    1. programs populating offline data should not become overwhelmed as volume of transactions increases.

    2. add programs populating offline data easily as volume increases

  6. We want to consider as many existing tactics, approaches, and solutions to integrating EOSIO chains with offline data analysis as possible.

  7. We desire to maintain, or increase FIO/EOSIO maintainability.

    1. make sure we use EOSIO tactics preferentially.

    2. always try to maintain merge-ability of FIO Core to and from EOSIO.

  8. We want to support all manner of offline analytics (grafana, and others).

  9. Verifiability – we must be able to verify that the offline data matches what is in state on the chain at any time.

  10. Authorization – we may need to consider additional authorization models to permit the update of data associated with state data stored off chain.

 

Existing FIO initiatives

This impacts some of our projects aimed at updating state, chain code, and aligning FIO with eosio code base to simplify future upgrades and integrations:

 

 

EOSIO initiatives and issues

GitHub - dfuse-io/dfuse-eosio: dfuse for EOSIO

https://eos.io/for-developers/build/demux/

Enhancement: ship data consistency proofs; block query API · Issue #22 · eosnetworkfoundation/mandel

 

Summary of present initiatives and issues

ship data consistency proofs – this would be very beneficial for the FIO off chain state database. We will keep an eye on this and try to meet with Todd Flemming to see if we can get an understanding of the level of interest within EOSIO regarding solving this.

Demux – this effort is perceived as lacking support and momentum.