Problem: Update architecture for FIP-1 to include the ability to page and cache data on wallets
One of the underlying issues in the FIO workflow is the lack of searching and querying of data based on time. This lack of functionality directly affects how the wallet partners show that data to their users. This proposal will attempt to address this issue with the use of a new index table.
The initial 1.2.0 Table migration is still recommended with this approach. It will significantly reduce the need for future migrations due to its member additions, and it also allows old requests to be bound into the newly proposed index table.
New table structure proposed (fio.request.obt):
| Code Block |
|---|
struct ledgerItems { |
...
vector<uint64_t> fio_request_ids; |
...
vector<uint64_t> recordobt_ids; |
...
} |
...
struct [[eosio::action]] reqledger { |
...
uint64_t account; |
...
ledgerItems transactions; |
...
uint64_t primary_key() const { return account; } |
...
} |
...
Example:
Account: d98asauia
...
During a new transaction, we’re able to remove records after reaching a designated limit. ( force use of history node ). If removal route is desired, we need to define limits. 50? 100? 1000? ( not right now ) Recommend to do during this release if history node is the recommended system for retrieving request data. Otherwise a new action call will be required to remove old request data for each account. The above example shows how removed data will still keep time integrity based on new available primary keys.
Proposed changes to chain_plugin:
Instead of iterating through ALL the records for paging, we’re able to utilize the new index table to assist in the offset and limit parameters. This allows us to now show the results from an earliest -> latest outline or vise versa, depending on the implementation.
Example:
offset: 2
limit: 2
Pseudocode:
| Code Block |
|---|
offset = 2; //default to 0 when initialized |
...
limit = 2; //if no limit, use vector size in production |
...
ledger_iter = reqledger.find(account); //retrieved from bind2eosio |
...
for( int i=0, i < limit, i++ ) { |
...
uint64_t req_id = ledger_iter->transactions.fio_request_ids[i+(offset-1)]; |
...
auto fioreqstss_iter = statusByRequestId.find(req_id); |
...
...
//Take information needed in the result and push back into the result vector. |
...
offset++; //or negate depending on the result approach |
...
} |
...
uint64_t more = ledger_iter->transactions.fio_request_ids.size() - ( offset + limit ); |
The “more” result is also greatly improved with this approach because a developer can take the size of the vector inside the transactions data structure and subtract the limit and offset.
...
This new contract update will be proposed via msig to the protocol. This will finalize all necessary changes to complete the transfer of FIO domains and addresses while maintaining request integrity.
Release Plan
Step 1 (fio.contracts Bravo-1)
After this update, new Request and OBT data coming in will be entered into both the old and new tables.
Step 1 introduces the new table structure and adds a new “temporary” action call called migrtrx. This action enables the top 21 block producers to start migrating existing data over to the new formatted index tables.
This will migrate the three current tables (fioreqctxts, recordobts, fioreqstss) into two new tables (fiotrxts, reqledgers). It also creates and uses a temporary table (migrledgers) that is used during the migration process.
Example:
./clio -u http://localhost:8889 push action -j fio.reqobt migrtrx '{"amount":"441","actor":"qbxn5zhw2ypw"}' -p qbxn5zhw2ypw@active
The maximum amount of records able to be transferred during a single transaction has been set to 10. Any input over this value will be set to max value. After all records have been migrated, the action will return back a success response.
It is preferable if a single BP takes this on and calls migrtrx until all data is migrated.
[FIP-1.b] Step 1: Add continuous migration of request and obt data (FIO #182)
The complete migration of data has to happen before fio chain is released.
fio chain release (Bravo)
Step 2 updates the getters (e.g., get_xyz_fio_requests)
Users pointing to non-upgraded nodes will be retrieving data from the old tables and users pointing to upgraded nodes will be retrieving data from the new tables.
Contract Step 2
Step 2 does two things:
Modifies the Request code to stop updating the old tables. After this release, only the new tables will get updated.
Modifies the
migrtrxcall. It will now remove data from the old tables.
Block producers are able to call migrtrx and remove up to 25 records during a single transaction. Added functionality also allows for the API endpoints to use the new data structure introduced in step 1.
[FIP-1.b] Step 2: Remove old data from index table (FIO #185)
All fio chain nodes should be upgrade prior to rolling out Contract Step 2.
Once the data has been deleted, wallets pointing to nodes that have not upgraded:
New OBT and Requests will be put into the new tables
Get requests and OBT getters will point to the old tables so they will get “no request found” or the OBT data will not be found results.
Contract Step 3
Step 3 includes:
The removal of
migrtrxtemp actionRemoves old table references (
fioreqctxts,recordobts,fioreqstss)
[FIP-1.b] Step 3: Finial Integration of request and obt data restructuring. (#183)
Wallets pointing to nodes that have not upgraded:
New OBT and Requests will be put into the new tables.
Get requests and OBT getters will get a 500 “Error 3060003: contract table query exception” error.