Simplify Development/Deployment of Read only Chain API endpoints

Overview --

This document will discuss the enhancement/redesign of getter/read operations provided by the FIO protocol. it will also discuss larger issues of state management in the FIO protocol.

 

Assumptions--

Familiarity with EOSIO and FIO design and implementation is assumed.

Background --

The FIO protocol is foundationally based on EOSIO.

FIO provides getters/readers of the chains state in 2 ways.

  1. get table - using get table in an HTTP post provides users the ability to see the contents of state tables by directly querying the tables.

    1. advantages – users can query table data any time they like, as long as the table exists on chain.

    2. disadvantages – users must know table structure, indexing models, and also they must implement the relevant business logic of the FIO protocol if they are to rely on the data that they query.

  2. chain plugin – the chain plugin provides an HTTP enabled query capability which consists of a set of end points (like get_fio_balance) which can contain business logic and return structured results.

    1. advantages – users use pre-defined endpoints with documented parameters and get structured results that have FIO business logic applied by the protocol.

    2. disadvantages – the chain plugin is integrated into the FIO protocol core code, so to get newly developed getters, users must upgrade an API node which they have to run, or they have to wait for the owners of the API node they use to upgrade, upgrades take time energy and integrations require resources to perform code level changes.

 

History/drivers –

With respect to getter/reader capabilities of FIO there are some well known deployment impacts of the present design approach.

Organizations like binance, and other integrators have some amount of inertia when it comes to performing upgrades to the FIO Core code, and who can blame them, we do frequent releases with lots of feature function and we are asking them to upgrade their API node quite alot and they have lots of other tokens they integrate and also their own business concerns that might be higher priority than always upgrading their FIO node. So this inertia should be avoided completely by the FIO protocol.

As a usability focused protocol we want to provide our integrators with an “easy” button when it comes to keeping up with the latest developments and features affecting the FIO protocol, especially the addition of new getters and modified getters for the various FIO projects that are delivered regularly.

In a larger sense we want to provide offline analytics and we want to enrich the information that can be associated with a FIO integrated transfer, or FIO request for funds.

 

EOSIO/ledger state limitations

There are some well known limitations of EOSIO and other blockchain technologies when it comes to providing rich offline analytics and tx associated data references. The use of state tables does not provide powerful query-ability. The transactional nature of the ledger limits the structure of the data provided for analytics purposes through history nodes.

We desire to provide a performant on chain solution for the ledger implementation. Along with this, we want to provide a rich and possibly varied set of offline relational data associated with transactions contained in the ledger.

EOSIO (and FIO) are designed with get/read capabilities as mentioned above in this document. We want to provide integrators a hassle free interface they can use to query the chain (not only for state information and history of the ledger), but for any data that might be associated with a FIO request, or a FIO integrated transfer).

 

Objective –

The objective of this effort is to explore strategies, and identify tactics that will provide the FIO protocol more “optionality” when it comes to the deployment and integration of the chains read/getter routines. We will consider if the use of off chain data is a strategy that might be most useful for this and other purposes. Each tactical option provides differing levels of ease and efficacy for integrators wishing to keep up with changes in the FIO protocol. Each option provides different risks and rewards of development.

We will provide a list of strategies for consideration.

We will provide a list of possible solutions/tactics for solving the getter/reader question contained herein.

We will explain the pertinent details of each option/tactic.

We will state the risks and benefits of each option.

We will provide biz dev and the planning committee enough information to consider and discuss these options objectively and determine what efforts might be most effective for funding going forward.

 

Possible strategies for addressing deployability of getters

S1 – Enhance the FIO Core code

  1. modify the FIO Core c++ to provide hot deployable getters, or even eliminate the need for deployment altogether by providing scripting in parameters.

S2 – Place getters into the FIO contracts layer

  1. make the getters to exist in the contracts layer, eliminate use of the tx log, and eliminatesome signing requirements for these tx.

S3 – Provide an off chain (relational) reflection of state that may contain supplemental information regarding ledger transactions.

 

Tactics/ Possible Solutions 

T1. Make getters hot deployable in chain plugin.

The idea here is to enhance the chain plugin so that the upgrade of the FIO Core for getter modifications becomes easier.

Overview

A table will be added to state in the FIO Core which identify by name the getter calls that can be performed. FIO system contracts will be enhanced to provide actions permitting BPs to update this information on chain.

Modifications will be made to the FIO chain plugin to provide a pluggable model of software extensibility for the chain plugin. The chain plugin will allow new dlls to be deployed quickly and atomically to a location on disk. Configuration information will be added to the chain plugin (either on chain or on disk) to identify the plugins to load. The chain plugin will be enhanced to provide a callable “update me” routine in the chain plugin to reinit the chain plugin with the latest updates. The chain plugin will be enhanced to provide a new endpoint in the chain plugin that uses json for parameters that define the getter to call and its parameters.

 

Benefits

This might ease deployment of the FIO Core code, by modularizing the components used to update getter routines, and by making a hot deployment of updates.

Risks

This might complicate deployment of api nodes (more information to manage and mis-configure, we need a process of updating the config info to be designed which complicates governance as well)

integration complexity increases, because this interface uses a meta data based set of parameters which further complicates calling getters.

 

T2 Modify chain plugin to provide a scripting style interface in its parameters.

The idea here is to enhance the chain plugin so that the upgrade of the FIO Core for getter modifications becomes easier.

Overview

A table will be added to state in the FIO Core which identify by name the getter calls that can be performed, along with the specific logic that is to be executed by each getter. FIO system contracts will be enhanced to provide actions permitting BPs to update this information on chain.

Modifications will be made to the FIO chain plugin to provide a new endpoint in the chain plugin that uses json for parameters that define the getter to call and its parameters.

A script interpreter will be introduced into the FIO Core, which will interpret the specific logic of each getter and execute this logic as part of the call to the getter.

Benefits

This effort removes deployment of getters as a concern from the API node operators and integrators.

Risks

This effort introduces a script interpreter into the FIO code base that is of equal complexity to the contracts layer.

Complexity

This is a very complex project with security, maintenance, testing, deployment, and other impacts to the FIO protocol for development and operations.

EOSIO compatibility

This is a significant enhancement to the EOSIO set of capabilities, so we will need to coordinate this and message and manage and possibly integrate this into the EOSIO community.

 

T3 Move all read/getter routines to FIO contract layer

The idea here is to make the deployment of getters to be “out of the hands of integrators” and “into the hands of the block producers”. in this model, the getters will be implemented in a set of getter specific contracts in the FIO protocol. Block producers will deploy updates as they do any contract modifications to FIO, via MSIG approval by a majority of the active BPs. All get routines will be ported into a set of contracts in the FIO protocol for use via signed contract calls.

 

Details –

We will design a set of getter contracts that will provide the “read portal” for the FIO protocol. We will add information into state regarding the getters that may be called. We will add a new action that provides users the ability to call a generic “read” action which will use json to identify what getter to call along with the necessary parameters.

Because this is a contract call, users will have to perform packaging of a signed transaction on the FIO protocol to call these getters. We may be able to limit this packaging to eliminate the need for private key signing. The core will be modified to “skip” adding these tx to the FIO block log (transaction logs). We will also mod the core to “skip” private key signing checks for the getter calls which are listed in the new state table. This alleviates the need to worry over account information for the getter calls (though other signed transaction details must be observed in the packaging of the call for the FIO chain).

 

 Benefits

this provides block producers the ability to control when where and how the chain is upgraded for all getters. this offloads the responsibility of upgrading the chain from integrators.

getters can be enhanced as fluidly as with any contract update to the FIO protocol

 

Risks

This effort introduces a set of getter actions which can bloat the contracts layer, and which require signing (but not the same signing as we use for other calls to the contracts).

 

Complexity

This is a very complex project with security, maintenance, testing, deployment, and other impacts to the FIO protocol for development and operations.

This effort increases integration complexity significantly, and will impact SDKs and all integrators of FIO.

EOSIO compatibility

This is a significant enhancement to the EOSIO set of capabilities, so we will need to coordinate this and message and manage and possibly integrate this into the EOSIO community.

This slightly different signing of the getter tx is cumbersome and confusing.

 

T4 Make a new app (fioget).

The idea here is to create a new application separate from the FIO Core which functions much like the chain plugin, but which leverages hot deployment capabilities mentioned above. we can consider to implement some smart upgrade capabilities (Think App Store or fully automatic style upgrades), by detaching this from the FIO Core, we can provide a get capability that is loosely coupled to the FIO Core and which has more options for upgrading.

This project is best summarized as “some form of item 2 above, but in a separate application from the FIO Core”

  1. We can provide meta data based query and hot deployment of upgrade components.

  2. this component can be run without an API node, but requires interface to an api node to query. this can be run from any device and becomes a building block for block explorers and other tools requiring business logic relating to all aspects of FIOs functions

Benefits --

loosely coupled from the FIO Core code, adds hot deployability and other features without impacting the core code of the FIO protocol.

Risks--

this causes duplication of getter logic (inside FIO Core and outside).

integrators still need to install and manage this new app much like they do the FIO Core, so this complicates deployment for integrators.

 

T5 ************** Provide complete off chain solution **************

The idea here is to provide a history node integrated, offline relational database that will contain the near real time reflection of the FIO state within it. We might provide option 1 above as the front end providing getter capabilities to FIO. We might provide an integration of existing EOSIO componentry or other technology solution available in the crypto space which will provide this.

if an off the shelf solution is not used, An ID/key will need to be designed and defined to provide cryptographically protected proof of authenticity for records contained in state. (to provide proof of authenticity for off chain data)

Benefits --

This provides near real time update of state information off the chain in a relational structure which can be powerfully queried using relational database technology.

This provides the foundation for simplifying and downsizing the FIO on chain state model, which is a key objective as the FIO protocol matures.

  Risks –

Unknowns of design must be discovered and documented --

we must address the following topics

choice of off the shelf solution, or design and implement.

governance of this off chain storage.

deployment image of this offline storage.

these topics will be hotly debated from a decentralization perspective and these decentralization concerns will complicate the process of design and development.

This will be a long lived multi phased project with complex deliverables which must address many yet to be fully identified issues

 

EOSIO/Community Considerations

For all of these options we may benefit from including experts in the EOSIO and crypto community as others are also heavily engaged on these and other questions pertaining to the limitations of ledger technolgies such as EOSIO and FIO.

 

Conclusions--

Analysis and discussion of these options leads us to the following conclusion.

Option 5 seems preferable as a choice for the FIO protocol.

Selection of this option allows the FIO protocol to address some key limitations of the present implementation of the FIO state model which are well known and well understood. The limitation is that state is VERY rich in its modeling and this rich state model becomes concerning as both volume of tx, and adoption rates increase for FIO.

Option 5 supports the evolution of the FIO protocol to provide a rich and varied set of orienting data associated with the transfers and requests which are performed on chain.

Option 5 provides rich and sustainable offline analytics updated in near real time. It also provides the foundation upon which the FIO state model can be simplified on chain. The size of the on chain state model can be reduced quite significantly, while providing a rich, and extensible set of data associated with the transactions that occur on chain.

Option 5 provides a scalable and standardized (and possibly DOA governed) off chain transaction data layer for the FIO protocol.

 

********* NEXT STEPS

It is recommended that we secure the general agreement of key thought leaders that the selection of option 5 as a solution is the preferred path for the FIO protocol.

If this is generally agreed, we must then vision, strategize and document the iterative deliveries that will surround the FIO Offline State Database (FOSD – pronounced FOZ-DEE).

We will proceed to scope and vision this set of projects that surround FOSD into something that can be presented to the FIO planning committee and then prioritize these efforts and deliver them iteratively, or as one set of Epics which result in the ultimate objective.

Going Forward --

This document must be reviewed by developers and community thought leaders, including EOSIO experts, and discussions must be had to find agreement on the preferred path forward. This path needs to be prioritized and then worked appropriately….

 

The scope of this spike has illuminated a possible need to focus on analysis and design possibilities for off chain state database solutions. This larger scope is reflected in the following wiki page.

https://fioprotocol.atlassian.net/wiki/spaces/DAO/pages/473268331

and is reflected in the following story.

https://fioprotocol.atlassian.net/browse/BD-3235