Release 3.5/2.9/1.9 Release Checklist

Owned by Michael Leet
Last updated: Nov 27, 2023
9 min read

his document is intended to assist release managers in tracking the progress of required work items for a given release. Lead developers can use this checklist to understand how they might assist the release manager in achieving a release of the FIO protocol.

Release Stakeholders

Whenever checklist items are initiated or completed, communications should be posted to the interested stakeholders so they can remain in the loop with respect to the progress of a release.

Marketing – marketing/marketing discord channel (tag ash).

Account management – internal/account management (Emily)

QA – core team private on discord (Eric and Ben)

BPs – Fio main net and fio test net telegram channel, Block-producers under discussion groups.

Core team – core team private channel

Community developers – FIO developers telegram channel

Product managers – product discord channel

 

Please refer to the field descriptions below the checklist for more information on each item.

Subject

Owner

Description

Jira Links and Notes

Subject

Owner

Description

Jira Links and Notes

Release planning

 

 

 

Update Master Release Schedule

PM

Update master release schedule, Master Release Plan , with all release info

Done

Create project wiki

PM

Create a project wiki created for release under Releases This will be used to store all relevant release documents.

Done - EB/Core

Release stories and subtasks

PM

Create and epic, stories, and subtasks for release. There should be a story for:

  • Release N.N Project Management

  • Release N.N QA

  • Release N.N Release Management

Subtasks should be created for each item in this checklist.

https://fioprotocol.atlassian.net/browse/BD-4550

Release kickoff meeting

PM

Set up a kickoff meeting with team to review release

Done - EB

Release script

Release

Create rollout/release script for this effort (added actions, removed actions, table migrations, order dependent detailed instructions on how to roll out these changes successfully. include all necessary MSIGs). This should include detailed instructions on how to verify that the rollout was performed successfully.

Refer to the Master Release Plan, Master Release Plan, for deliverables to capture.

Refer to FIPs Repo, https://github.com/fioprotocol/fips , for actions, fees, etc., to capture in release script

Once complete, review the rollout plan and verification with architecture and QA/Integration teams.

Release script template: [Release n.n] Release Script

See example: [fio 3.3 fio.contracts 2.7] Release script

Done

Security

 

 

 

Code audit

PM

A security focused audit of contract modifications should be strongly considered for any new actions, or any modifications to actions that involve changes in authorizations, or error logic. Code audits can be performed by core team members, or they can be performed by third parties specializing in EOSIO security issues.

Findings for security audits are sensitive and may be published to the public after careful consideration of the consequences of publishing this material. Once vulnerabilities have been adequately addressed all findings and remediations should be posted to the wiki for the project in question. Provide a set of links to completed items, or provide a point of contact for your project if items are in work.

Security documentation: https://fioprotocol.atlassian.net/wiki/spaces/FO/pages/141262894

 

Devnet

 

 

 

DevNet Smoke Testing

Release

Complete full devnet fio.test testing

Done

DevNet Fork Testing

Release

Complete full devnet fio fork testing

See: https://developers.fioprotocol.io/docs/developers/devnet

This task has two flavors:

  • To test a forking change, and that processing executes as normal before and after the (DevNet customized) timestamp

  • To verify that no forking occurs in any BP node (standard DevNet testing)

Done

DevNet - Performance/Scalability testing

Release

Perform DevNet performance and scalability tests

Performance and Scalability testing is a responsibility of the Developer and QA, however, it may be necessary to identify release level performance and scaling requirements

 

DevNet - QA testing

Release

fio.test regression tests completed and run cleanly on Devnet

Done

LocalNet/DevNet

 

 

 

*Net Release rollout test

Release

Validate release script, updated by developers for each feature, detailing all addaction, createfee, and new and existing contract updates

Perform local (single machine) test to validate release script

Update the Rollout Plan with any new information

See: https://developers.fioprotocol.io/docs/developers/devnet#contract-testing-fiocontracts

See: https://developers.fioprotocol.io/docs/developers/devnet#local-testing

Done

*Net Performance and Scaling

Release

Verify performance and scalability tests run on LocalNet (and results added to Dev Spec)

Identify performance and scalability tests to run on DevNet

 

SDK Pre Release

 

Changes to Typescript SDK may be necessary for each release of the FIO protocol. SDK changes should go through a QA process. The release manager must integrate the SDK changes required for a release, then perform acceptance tests on these changes.

 

SDK release story

PM

Create story (if not already done) to track SDK Release

Tracked as release for AlephOne team: https://fioprotocol.atlassian.net/projects/DASH/versions/10081/tab/release-report-all-issues

SDK dev complete

PM

Confirm that SDK development is complete for all features in release

Make sure the package.json in fiosdk_typescript has the correct version

Eric: https://fioprotocol.atlassian.net/browse/BD-4542

SDK release candidate

Release

Create release/n.n.x branch (where n=contract release number)

Create pre-release (Release Candidate - FIO Typescript SDK x.x.x-rc1)

TODO

Testnet - Setup & QA

 

 

 

fio.test pre-release

Release

  • Create release/n.n.x_m.m.x branch (where n=contract release number and m=chain release number)

  • Confirm that the package.json references the new kiosk_typescript branch

Done.

SDK Update fio.test to point to SDK release branch

Release

Update fio.test package.json file to point to the SDK pre-release.

Review as sdk was in-progress

fio.devtools pre-release

Release

Create release/n.n.x_m.m.x branch (where n=contract release number and m=chain release number)

Done

fio.contracts pre-release

Release

  • Create release/n.n.x branch (where n=contract release number)

  • Create pre-release (Testnet Release Candidate - FIO Contracts x.x.x-rc1)

  • Update release notes

  • Create PR for hashes on fio.mainnet > releases-testnet.md

Done

See https://github.com/fioprotocol/fio.contracts/releases/tag/2.9.0-rc1

fio pre-release

Release

  • Create release/m.m.x branch (where m=chain release number)

  • Create pre-release (Testnet Release Candidate - FIO x.x.x-rc1)

  • Update release notes

  • Replay test by BP (needs to be defined and documented

  • Track chain upgrade of BPs

Done

See https://github.com/fioprotocol/fio/releases/tag/3.5.0-rc1

 

 

Testnet - Release branch testing

QA

After all release branches have been created for new release:

  • Run full fio.test regression, including tests that require manual updating of the contracts

Done

Testnet - Manual History node testing

QA

For transactions that automatically create accounts (e.g., regdomain) if the public key used is new, we need to make sure that the transaction shows up in bloks.io transaction list.

TODO: Review this requirement…

Testnet Release

 

 

 

Release msigs

Release

  • BP to create msig to set addaction and createfee for new actions

  • BP to create msigs for updated and new contracts

  • Review msigs

  • BP to post msigs to Testnet

Done.

Phase 1: https://t.me/fiotestnetb/9988, https://t.me/fiotestnetb/9989, https://t.me/fiotestnetb/9990

Phase 2: https://t.me/fiotestnetb/10136

Phase 3: https://t.me/fiotestnetb/10343

Communication

Release

  • Socialize changes with all BPs on Testnet and the community and marketing

  • Monitor and coordinate the rollout with BPs

Done

Testnet - Release validation

Release

  • Confirm hashes of testnet

  • Confirm ABIs deployed correctly (using ./clio get abi)

  • Confirm createfee added fees with correct endpoint and type

  • Confirm addaction added correct actions with correct contracts

Actions. fees validation successful
Hashes, ABI validation failed, fio.system correct, all others (fio.address, fio.escrow, etc) failed.

Testnet - Smoketest

QA

Update fio.test smoketest with new actions and getters and run against Testnet

Executed; failures in fio.perms tests.

Testnet - Manual Feature validation

QA

Perform manual validation for all actions and getters listed in the Master Release Plan.

Also work with developer to do any release validation testing noted in the Deployment and Rollout guides.

Chain validation shows fio.perms set contract failed. Failures also occurred in other set contracts.
Lesson Learned: use postman get_block to validate msig block (block+1, etc.)

Testnet - Replay test

Release

Perform full replay test (needs to be defined and documented)

 

Testnet Node*

Release

Update FIO Testnet node

See: https://developers.fioprotocol.io/docs/chain/node-build

TODO

Mainnet - Setup

 

 

 

Mainnet prep

Release

  • Create upgrade checklist of BPs and integration partners

  • Work with account management to coordinate Mainnet rollout plan with the BP, wallet, and exchange community, watch over the execution and help to ensure rollout is completed in full.

  • Socialize release changes with all BPs and the community and marketing

 

fio.test Mainnet setup

Release

  • Merge release/n.n.x_m.m.x branch to master, create vn.n.n_m.m.m tag

  • Run test tag against new release

 

fio.devtools Mainnet setup

Release

  • Merge release/n.n.x_m.m.x branch to master, create vn.n.n_m.m.m tag

  • Update base contracts to current mainnet contracts

  • Update readme to show new version changes on fio

  • Merge to master and cut a new tag/release

 

fio.contract Mainnet setup

Release

  • Merge release/n.n.x branch to master, create vn.n.n tag

  • Move pre-release to release (Release - FIO Contracts x.x.x)

  • Create PR for hashes on fio.mainnet

  • Encourage BPs to vote on new endpoint fee

  • msig fio.contracts release

  • Perform the rollout verification and report findings

  • Post msig links to main FIO Telegram channel so member can track progress

 

fio Mainnet setup

Release

  • Merge release/m.m.x branch to master, create vm.m.m tag

  • Move pre-release to release (Release - FIO x.x.x), creating the GitHub Release with new functionality, fixes, etc. documented.

  • Build the FIO repo (use -P option) and Generate .deb and .tgz build files using fio.package

  • Upload .deb and .tgz build files to;

  • FIO Releases page for this release

  • AWS > S3 > fioprotocol > Mainnet

  • Update the FIO.docker repo (package release links in dockerfile) and create pull request notifying Todd of the new release artifacts

  • BP mainnet node upgrade checklist is complete (BP spreadsheet, or create one in Jira)

 

Mainnet - Release

 

 

 

addaction and createfee msigs

Release

  • msig to set addaction and createfee for new actions

  • Encourage BPs to vote on new endpoint fee

Example:

createfee '{"end_point":"transfer_tokens_fio_add","type":"1","suf_amount":"958695652"}'

addaction '{"action":"trnsloctoks","contract":"fio.token","actor":"eosio"}'

 

Contract msigs

Release

  • BP to create msigs for updated and new contracts

  • Review msigs

  • BP to post msigs to Mainnet

 

Mainnet - Release validation

Release

  • Confirm hashes of mainnet

  • Confirm versions of mainnet nodes (chain)

  • Confirm ABIs deployed correctly (using ./clio get abi)

  • Confirm createfee added fees with correct endpoint and type

  • Confirm addaction added correct actions with correct contracts

 

Mainnet - Manual QA validation

QA

Perform manual validation for all actions and getters listed in the Master Release Plan.

Also work with developer to do any release validation testing noted in the Deployment and Rollout guides.

 

Post-deployment

 

 

Mainnet validation - fio chain

Release

 

Update FIO Hosted API Nodes

Release

  • Upgrade Registration site API nodes (2 load balanced nodes)

  • Notify Todd to upgrade Analytics API node (single State History node)

 

FIP status

PM

Move FIP status to Final and include release links; update master README index

 

Devhub

PM

 

fiosdk_typescript-examples

PM

Add examples for new actions to fiosdk_typescript-examples repo

 

Partner upgrades

PM

  • Create upgrade blog post and send to Account Management team to manage upgrades

  • Work with Account Management to coordinate Mainnet rollout plan with the BP, wallet, and exchange community, watch over the execution and help to ensure rollout is completed in full.

 

SDK Production Release

 

 

 

Typescript SDK Release

Release

Tracks the release of the latest SDK to production.

  • Code review all changes

  • System test SDK release tag version against latest production releases (master branches) of fio, fio.contracts, fio.devtools, fio.test (If updates to test are needed, confirm fixes are put into the fio.test release/vn.n.n release branch and the develop branch (i.e., merge fixes from release branch))

  • Merge develop to master and create production release tag from master

  • Typescript: Publish new SDK release on NPM (see: https://github.com/fioprotocol/fiosdk_typescript#publishing-to-node-package-manager-npm)

  • Work with Account Management to socialize the release to integration partners

 

 

 

Process Description --

Each item that is to be addressed in the checklist should have a story associate with it under the release epic in jira. also there should be a release section of the wiki which contains all of the artifacts generated for a given release, and under this wiki should also be contained all the details of the release planning and execution for future reference of the community.

 

Devnet testing

Background – In order for a release of the FIO protocol to be rolled out on test net and main net, the release manager must first assemble the set of commands that will upgrade the current main net version to become the new release. The commands may include adding actions to FIOs list of allowed contract actions, adding/modifying the fees associated with FIO actions, creating new FIO system accounts and contracts, and updating all of the required FIO contracts with the changes for the new release. The release manager must assemble a complete list of the required commands, and execute them on a private test network, then perform validations of the resulting release. The commands, and test results must be posted on the wiki for this release.

Objective – The objective of dev net testing is to assemble the list of commands required to upgrade FIO for this release, and to complete acceptance testing of a local private FIO network upgraded using these commands.

Tactics – on dev net, build the release version of the FIO Core code and the FIO contracts, copy these off to the side. Then build the dev net using the Main net versions of the contracts and core, and spin up the dev net. Perform all of the necessary clio commands to upgrade the contracts. once this is completed run a regression test to ensure all chain operations function as expected. if there are significant changes to the FIO Core code which result in special logic and timing regarding the core and contracts rollout (this is called a forking change), then special test plans must be made and executed in accordance with the needs of the releases unique needs. If no forking logic is being released then acceptance and QA tests and some performance tests will be performed and published onto the wiki

Deliverables – The release manager will post on the wiki the commands for this release, and they will also post on the wiki the test results for the tests performed.

Contract rollout commands – The release commands for each project included in the release should be a part of the rollout plan for each project included in the release, assemble the commands for the release into one list of commands, and fill any missing gaps by cooperating with the necessary leads if commands have not yet been provided. the following actions are commonly required --

Add action – add an action to the permitted list of actions in the FIO protocol

Add Fee – add a new fee for a new action in the FIO protocol.

Add new system account – When new contracts are added to FIO an owning account must be created.

Set system account authorizations – set permissions and authorizations on new system accounts

Set privileged.

update authorizations

Set Contract – set a new contract on the FIO protocol, or update an existing contract.

Set ABI – set a new ABI on the FIO protocol, or update an existing ABI.

Example:  https://fioprotocol.atlassian.net/wiki/spaces/FD/pages/471040008/New+contract+deployment

See also

 https://developers.fioprotocol.io/docs/developers/devnet

 

Contract testing – After contracts have been upgraded, acceptance testing must be completed, and findings must be published to the wiki.

Fork testing – If there are changes to the protocol which require backwards compatibility considerations then forking tests must be performed, these test require code changes to be made before the fork testing to ensure that the forking logic works equivalently in the dev net environment. Results and findings must be published to the wiki upon completion.

Performance testing – Some projects have dev net level performance test considerations, when a release includes the need for performance testing then tests must be run and findings published to the wiki for the release.

QA testing – QA regression tests should be completed on each release before dev net testing, but full regression on dev net only increases our confidence in a release. findings should be published to the wiki.

 

 

NOTE – other descriptions may be added if this is seen as helpful for the document