[FIP-1.b] OBT Optimizations - Development Spec

Owned by Casey Gardiner
Last updated: Jan 07, 2021 by Eric Butz
9 min read

Links

Design

Overview

Problem: Update architecture for FIP-1 to include the ability to page and cache data on wallets

One of the underlying issues in the FIO workflow is the lack of searching and querying of data based on time. This lack of functionality directly affects how the wallet partners show that data to their users. This proposal will attempt to address this issue with the use of a new index table.

The initial 1.2.0 Table migration is still recommended with this approach. It will significantly reduce the need for future migrations due to its member additions, and it also allows old requests to be bound into the newly proposed index table.

New table structure proposed (fio.request.obt):

struct ledgerItems {    vector<uint64_t> fio_request_ids;    vector<uint64_t> recordobt_ids; } struct [[eosio::action]] reqledger {    uint64_t account;    ledgerItems transactions;    uint64_t primary_key() const { return account; } }

Example:   

 Account:     d98asauia

 Transactions:     fio_request_ids[6,7,1, 55]

                recordobt_ids[13,2,544,23232] 

The requests contract will create a new entry if an account is not found during a request or recording an obt transaction. All sent and received records will be placed at the end of the transactions data structure to keep track of time using the id from available_primary_key on the action. This ordering has no impact on the performance of a plugin and will be able to parse the result data in either format. 

During a new transaction, we’re able to remove records after reaching a designated limit. ( force use of history node ). If removal route is desired, we need to define limits. 50? 100? 1000? ( not right now ) Recommend to do during this release if history node is the recommended system for retrieving request data. Otherwise a new action call will be required to remove old request data for each account. The above example shows how removed data will still keep time integrity based on new available primary keys.

Proposed changes to chain_plugin:

Instead of iterating through ALL the records for paging, we’re able to utilize the new index table to assist in the offset and limit parameters. This allows us to now show the results from an earliest -> latest outline or vise versa, depending on the implementation.

Example: 

  • offset: 2 

  • limit: 2

Pseudocode: 

offset = 2; //default to 0 when initialized limit = 2; //if no limit, use vector size in production  ledger_iter = reqledger.find(account); //retrieved from bind2eosio for( int i=0, i < limit, i++ ) {    uint64_t req_id = ledger_iter->transactions.fio_request_ids[i+(offset-1)];    auto fioreqstss_iter = statusByRequestId.find(req_id);    //Take information needed in the result and push back into the result vector.       offset++; //or negate depending on the result approach  } uint64_t  more = ledger_iter->transactions.fio_request_ids.size()  - ( offset + limit  );

The “more” result is also greatly improved with this approach because a developer can take the size of the vector inside the transactions data structure and subtract the limit and offset.

Important to note that data can be retrieved from an API method by either “earliest to latest“ or “latest to earliest”, whichever will make for most effortless development. Pawel?

MAS-1642 -( Modify search logic for get_obt_data, get_pending_fio_requests, get_sent_fio_requests )

The current table structure for FIO requests does not support public key searching. This results in improper requests returned after transferring an FIO address. To achieve this, a new table needs to be created, prior data needs to be migrated, and the old table contents need to be removed.


Current Table Structure Example:


The following index tables will need to be duplicated:

  • fioreqctxt ( fioreqctxts )

  • recordobt_info ( recordobts )


The major index table changes are the following:

  • Removal of payee and payer time

  • Removal of duplicate hex values.

  • Add support for key index searching


Purposed Table Structure Example ( Future/Binary extensions not shown )


Purposed unused variables for later use ( data will be initialized as null on creation ):

  • string futureuse_string1

  • string futureuse_string2

  • uint64_t futureuse_uint64

  • uint128_t futureuse_uint128

Once the new table is created, changes will be made to the following actions to bind new data  to the new index table:

  • newfundsreq

  • recordobt

The chain_api_plugin will need to be edited to allow for the searching of the public key supplied in the expression for the following API methods:

  • get_pending_fio_requests

  • get_sent_fio_requests

  • get_obt_data


Once the table names have been swapped in the action calls, a migration action call needs to be created inside the fio.request.obt contract. This action will accomplish the following:  

  • Allow only one actor the privilege to call action

  • Iterate from the beginning of the old index table

  • Take data and record it onto the new index table

  • Remove the old entry

  • Only allow 10-20 records to be processed during a single call


Best practice at this point would be to lock all requests during this time with a boolean check. At this point a proposed msig will be created with the above updates. It is recommended that the BPs get together via conference call to assist so the downtime during migration is limited. The migration action will be called until all records have been successfully transferred. Once all records have been migrated, the action will be removed from the contract, and the old table will have the following data member removals:

 


Another possible fix is to migrate the data back to the old table with changes to the old structure. 

This new contract update will be proposed via msig to the protocol. This will finalize all necessary changes to complete the transfer of FIO domains and addresses while maintaining request integrity. 

New Structure Notes 11/24/20 -

During the development of the above structure, some limitations arose that would hinder the scaling of account with large amounts of requests and obt actions. The following changes were made to better allow for searching payer and payee by status and account name.

  • Removal of the Ledger

    • The purpose of this ledger was to quickly search ids but this limited CPU processing time for high volume users when needing to append or modify the vector container due to serialization.

  • Additional searchable indexes inside the fiotrxts table.

    • These index searches allow for multiple queriable conditions inside a single, bit shifted operation. This also allows future development to search by status(es).

    • New global data member, STATUS_MULTIPLIER is used for shifting operations.

  • New time index search inside fiotrxts table.

    • This will allow time-based state cleanup in the future.

SDK Requirements

  • Note functional updates needed for SDKs

Functional Testing

Token Transfer

The following tests are focused on Token Transfer:

  • transfer-tokens.js

Plan: create new record-obt-data.js test:

  • Test single recordobt with bundles

    • Send FIO with Memo

    • Confirm get_obt_data returns correct record for payee and payer

    • Confirm content can be decrypted by payee

      • Include correct content for each content field and confirm it can be read by payee

    • Confirm status field for payee and payer

    • Confirm 2 bundles are deducted

  • Test single recordobt with no bundles

    • Same test as above, but with no bundles remaining

  • Error testing

    • Send with large content field that is too large

    • record obt with:

      • badly formatted payer_fio_address and payee_fio_address

      • Send character string to fio_request_id

      • payee_fio_public_key that does not own the the payee_fio_address

  • Paging testing

    •  

FIO Requests

The following tests are focused on FIO Requests:

  • fio-request.js

  • cancel-funds-request.js

  • paging.js > C. get_cancelled_fio_requests paging

Plan:

  • Merge cancel-funds-request.js and paging.js > C. get_cancelled_fio_requests paging into fio-request.js

  • Add new tests to fio-request.js

New tests:

  • FIO Request new tests

    • Create requests and get them into all status types

    • Confirm getter calls return correct information

  • Paging testing:

    • get_pending_fio_requests

    • get_sent_fio_requests

    • get_received_fio_requests (pull in the tests that Ed wrote)

Results

  • Link to JS tests and description of which test sections were updated/added

  • Paste results of tests

Performance Testing

Design

  • OBT

    • Update chain with 5000 OBT requests

    • Confirm fio.test (transfer-tokens.js, record-obt-data.js) runs cleanly

  • FIO Request - General

    • Update chain with 5000 requests of each status each from a different user

    • Confirm fio.test (fio-request.js) runs cleanly

  • FIO Request

    • Update chain with 5000 requests of each status for a pair of users (payer and payee)

    • Confirm that both payer and payee are able to retrieve all requests.

Results

  • Summary of performance tests that were run and results

Fork Testing Plan

This is not a forking change since users on current chain code and new chain code will maintain the same state. What changes in the new chain code is how the FIO Request getters work. In the current code, they call into the current tables. In the new chain code, they will call into the new tables. So, wallets that point to an API that has not upgraded will eventually get null data once we fully transfer over to the new tables and delete the old ones. We will maintain both tables for a time, but at some point we will do a contact msig that deletes the old table. So, we do not need a date in the chain code, the break will come when we do the msig.

Rollout/Release Plan

Contracts Release 1 - fio.contracts (Bravo)

After this update and the initial call to migrtrx, new Request and OBT data coming in will be entered into both the old and new tables.

Step 1 introduces the new table structure and adds a new “temporary” action call called migrtrx. This action enables the top 21 block producers to start migrating existing data over to the new formatted index tables.

This will migrate the three current tables (fioreqctxts, recordobts, fioreqstss) into two new tables (fiotrxts). It also creates and uses a temporary table (migrledgers) that is used during the migration process.

Example:

./clio -u http://localhost:8889 push action -j fio.reqobt migrtrx '{"amount":"441","actor":"qbxn5zhw2ypw"}' -p qbxn5zhw2ypw@active

The maximum amount of records able to be transferred during a single transaction has been set to 10. Any input over this value will be set to max value. After all records have been migrated, the action will return back a success response.

It is preferable if a single BP takes this on and calls migrtrx until all data is migrated.

[FIP-1.b] Step 1: Add continuous migration of request and obt data (FIO #182)

The complete migration of data has to happen before fio chain is released. An off chain validation should be done to confirm this.

Note that the partial implementation of transfer address (https://fioprotocol.atlassian.net/browse/BD-2232 ) will work after Step 1.

Contracts Release 1.b

Do the migration process:

  • Call migrtrx repeatedly

  • After the first call, new requests and OBTs should be added to both tables. (Test)

  • migrtrx is also used to check status updates.

Created test for initial migration: fio.test bravo-migr-test.js

Chain Release 1 - fio chain (Bahamas)

Step 2 updates the getters (e.g., get_xyz_fio_requests)

Users pointing to non-upgraded nodes will be retrieving data from the old tables and users pointing to upgraded nodes will be retrieving data from the new tables.

Contracts Release Part 2 - fio.contracts (migr/part2)

All fio chain nodes should be upgraded prior to rolling out Contract Step 2.

Step 2 does two things:

  • Modifies the Request code to stop updating the old tables. After this release, only the new tables will get updated.

  • Modifies the  migrtrx call. It will now remove data from the old tables.

Block producers are able to call migrtrx and remove up to 25 records during a single transaction. Added functionality also allows for the API endpoints to use the new data structure introduced in step 1.

[FIP-1.b] Step 2: Remove old data from index table (FIO #185) 

Once the data has been deleted, wallets pointing to nodes that have not upgraded:

  • New OBT and Requests will be put into the new tables

  • Get requests and OBT getters will point to the old tables so they will get “no request found” or the OBT data will not be found results.

Contracts Release Part 3 - fio.contracts (migr/part3-final)

Step 3 includes:

  • The removal of migrtrx temp action

  • Removes old table references (fioreqctxts, recordobts, fioreqstss)

[FIP-1.b] Step 3: Finial Integration of request and obt data restructuring. (#183)

Wallets pointing to nodes that have not upgraded:

  • New OBT and Requests will be put into the new tables.

  • Get requests and OBT getters will get a 500 “Error 3060003: contract table query exception” error.

Release Verification Plan

@Casey Gardiner Need this. Add info on your scripts that will verify data, etc.