Versions Compared

Old Version 4

changes.mady.by.user Casey Gardiner

Saved on

compared with

New Version Current

changes.mady.by.user Eric Butz

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

...

...

...

...

...

...

Table of Contents

Links

Design

Overview

Problem: Update architecture for FIP-1 to include the ability to page and cache data on wallets

One of the underlying issues in the FIO workflow is the lack of searching and querying of data based on time. This lack of functionality directly affects how the wallet partners show that data to their users. This proposal will attempt to address this issue with the use of a new index table.

The initial 1.2.0 Table migration is still recommended with this approach. It will significantly reduce the need for future migrations due to its member additions, and it also allows old requests to be bound into the newly proposed index table.

New table structure proposed (fio.request.obt):

Code Block
struct ledgerItems {
   vector<uint64_t> fio_request_ids;
   vector<uint64_t> recordobt_ids;
}

struct [[eosio::action]] reqledger {
   uint64_t account;
   ledgerItems transactions;
   uint64_t primary_key() const { return account; }
}

...

During a new transaction, we’re able to remove records after reaching a designated limit. ( force use of history node ). If removal route is desired, we need to define limits. 50? 100? 1000? ( not right now ) Recommend to do during this release if history node is the recommended system for retrieving request data. Otherwise a new action call will be required to remove old request data for each account. The above example shows how removed data will still keep time integrity based on new available primary keys.

Proposed changes to chain_plugin:

Instead of iterating through ALL the records for paging, we’re able to utilize the new index table to assist in the offset and limit parameters. This allows us to now show the results from an earliest -> latest outline or vise versa, depending on the implementation.

...

Important to note that data can be retrieved from an API method by either “earliest to latest“ or “latest to earliest”, whichever will make for most effortless development. Pawel?

MAS-1642 -( Modify search logic for get_obt_data, get_pending_fio_requests, get_sent_fio_requests )

The current table structure for FIO requests does not support public key searching. This results in improper requests returned after transferring an FIO address. To achieve this, a new table needs to be created, prior data needs to be migrated, and the old table contents need to be removed.

...

This new contract update will be proposed via msig to the protocol. This will finalize all necessary changes to complete the transfer of FIO domains and addresses while maintaining request integrity. 

New Structure Notes 11/24/20 -

During the development of the above structure, some limitations arose that would hinder the scaling of account with large amounts of requests and obt actions. The following changes were made to better allow for searching payer and payee by status and account name.

  • Removal of the Ledger

    • The purpose of this ledger was to quickly search ids but this limited CPU processing time for high volume users when needing to append or modify the vector container due to serialization.

  • Additional searchable indexes inside the fiotrxts table.

    • These index searches allow for multiple queriable conditions inside a single, bit shifted operation. This also allows future development to search by status(es).

    • New global data member, STATUS_MULTIPLIER is used for shifting operations.

  • New time index search inside fiotrxts table.

    • This will allow time-based state cleanup in the future.

SDK Requirements

  • Note functional updates needed for SDKs

Functional Testing

Token Transfer

The following tests are focused on Token Transfer:

  • transfer-tokens.js

Plan: create new record-obt-data.

...

Migration Release Plan

js test:

  • Test single recordobt with bundles

    • Send FIO with Memo

    • Confirm get_obt_data returns correct record for payee and payer

    • Confirm content can be decrypted by payee

      • Include correct content for each content field and confirm it can be read by payee

    • Confirm status field for payee and payer

    • Confirm 2 bundles are deducted

  • Test single recordobt with no bundles

    • Same test as above, but with no bundles remaining

  • Error testing

    • Send with large content field that is too large

    • record obt with:

      • badly formatted payer_fio_address and payee_fio_address

      • Send character string to fio_request_id

      • payee_fio_public_key that does not own the the payee_fio_address

  • Paging testing

FIO Requests

The following tests are focused on FIO Requests:

  • fio-request.js

  • cancel-funds-request.js

  • paging.js > C. get_cancelled_fio_requests paging

Plan:

  • Merge cancel-funds-request.js and paging.js > C. get_cancelled_fio_requests paging into fio-request.js

  • Add new tests to fio-request.js

New tests:

  • FIO Request new tests

    • Create requests and get them into all status types

    • Confirm getter calls return correct information

  • Paging testing:

    • get_pending_fio_requests

    • get_sent_fio_requests

    • get_received_fio_requests (pull in the tests that Ed wrote)

Results

  • Link to JS tests and description of which test sections were updated/added

  • Paste results of tests

Performance Testing

Design

  • OBT

    • Update chain with 5000 OBT requests

    • Confirm fio.test (transfer-tokens.js, record-obt-data.js) runs cleanly

  • FIO Request - General

    • Update chain with 5000 requests of each status each from a different user

    • Confirm fio.test (fio-request.js) runs cleanly

  • FIO Request

    • Update chain with 5000 requests of each status for a pair of users (payer and payee)

    • Confirm that both payer and payee are able to retrieve all requests.

Results

  • Summary of performance tests that were run and results

Fork Testing Plan

This is not a forking change since users on current chain code and new chain code will maintain the same state. What changes in the new chain code is how the FIO Request getters work. In the current code, they call into the current tables. In the new chain code, they will call into the new tables. So, wallets that point to an API that has not upgraded will eventually get null data once we fully transfer over to the new tables and delete the old ones. We will maintain both tables for a time, but at some point we will do a contact msig that deletes the old table. So, we do not need a date in the chain code, the break will come when we do the msig.

Rollout/Release Plan

Contracts Release 1 - fio.contracts (Bravo

...

)

After this update and the initial call to migrtrx, new Request and OBT data coming in will be entered into both the old and new tables.

Step 1 introduces the new table structure and adds a new “temporary” action call called migrtrx. This action enables the top 21 block producers to start migrating existing data over to the new formatted index tables.

This will migrate the three current tables (fioreqctxts, recordobts, fioreqstss) into two new tables (fiotrxts, reqledgers). It also creates and uses a temporary table (migrledgers) that is used during the migration process.

...

[FIP-1.b] Step 1: Add continuous migration of request and obt data (FIO #182)

The complete migration of data has to happen before fio chain is released. An off chain validation should be done to confirm this.

...

Note that the partial implementation of transfer address (https://fioprotocol.atlassian.net/browse/BD-2232 ) will work after Step 1.

Contracts Release 1.b

Do the migration process:

  • Call migrtrx repeatedly

  • After the first call, new requests and OBTs should be added to both tables. (Test)

  • migrtrx is also used to check status updates.

Created test for initial migration: fio.test bravo-migr-test.js

Chain Release 1 - fio chain (

...

Bahamas)

Step 2 updates the getters (e.g., get_xyz_fio_requests)

Users pointing to non-upgraded nodes will be retrieving data from the old tables and users pointing to upgraded nodes will be retrieving data from the new tables.

Contracts Release

...

Part 2 - fio.contracts (

...

migr/part2)

All fio chain nodes should be upgraded prior to rolling out Contract Step 2.

Step 2 does two things:

  • Modifies the Request code to stop updating the old tables. After this release, only the new tables will get updated.

  • Modifies the  migrtrx call. It will now remove data from the old tables.

...

  • New OBT and Requests will be put into the new tables

  • Get requests and OBT getters will point to the old tables so they will get “no request found” or the OBT data will not be found results.

Contracts Release

...

Part 3 - fio.contracts (

...

migr/part3-

...

final)

Step 3 includes:

  • The removal of migrtrx temp action

  • Removes old table references (fioreqctxts, recordobts, fioreqstss)

...

  • New OBT and Requests will be put into the new tables.

  • Get requests and OBT getters will get a 500 “Error 3060003: contract table query exception” error.

Release Verification Plan

Casey Gardiner Need this. Add info on your scripts that will verify data, etc.