Defines the projects, tasks, and KPIs associated with building and maintaining the FIO QA. FIO QA is mainly focused on automated test coverage and is less concerned with defining manual test cases that can be followed by a QA engineer.Chain.
Overview
There are several categories of FIO QA:
...
fio and fio.contracts automated testing
...
fio.test javascript tests
...
Devnet release testing
...
Testnet release testing
...
Mainnet release testing
...
Application testing
...
Includes manual and automated testing of FIO applications:
Dashboard
Registration Site
...
SDK testing
...
Running of SDK unit tests for every release.
...
projects associated with improving and maintaining the FIO Chain:
Core chain and contract development to support FIO Improvement Proposals (FIPs)
QA of core chain and contract bug fixes and new features
Release management of core chain and contract updates
Ensuring the security of the FIO chain
Maintaining compatibility with and contributing to the EOSIO ecosystem
Building a developer community around FIO
Core chain and contract development
QA
fio and fio.contracts automated testing
fio and fio.contract QA is covered by the fio.test javascript test harness. New automated tests are created for every new action, endpoint and feature added to the protocol. Full regression tests are run for every release.
...
All features, especially Mainnet features, do not have full regression tests
Testing lags development and impacts releases
Limited QA resources to develop automated tests
No time to focus on test harness architecture
QA not integrated into CI pipeline
Devnet release testing
Devnet release testing is currently done by the development team. It involves deployment and upgrade testing of fio and fio.contracts.
Testnet release testing
Testnet release testing currently consists of:
Running an automated smoketest against Testnet before and after deployment
Some manual testing of Testnet using Cryptonym
Some manual testing of features by development and product teams
Mainnet release testing
Mainnet release testing currently consists of:
Some manual testing of Mainnet using Cryptonym
Some manual testing of features by development and product teams
Application testing
Application testing is currently covered by manual UAT testing by the development and product teams. Options being considered are:
Using Proof’d automated regression scans to catch changes
Writing Web UI automated tests
Increasing manual testing
SDK testing
SDK testing currently consists of:
...
SDK unit tests written by development team
...
Typescript unit tests run against new releases of fiosdk_typescript and new fio and fio.contracts releases
...
Kotlin and Swift unit tests are not currently maintained or run for new releases.
...
Release Management
Security
EOSIO compatibility and community support
Resource estimates
1 FTE for updating test harness and overseeing QA automated test development
1 FTE for adding missing regression tests (can be done by multiple developers)
Bounties or outsourcing?
1.5 FTE for adding new regression tests for new features (2:1 ratio of developers to QA)
0.25 FTE to maintain Proof’d or other automated test suite.
0.2 FTE for release testing
0.1 FTE for SDK release testing (does not include SDK unit test development)
...
KPIs
There are several categories of FIO QA:
Task | Description | KPIs |
|---|---|---|
fio and fio.contracts new feature | Writing automated tests for new fio and fio.contracts features Core chain and contract development |
|
QA | Testnet release testing | Manage all manual and automated Testnet release testing
|
Backlog of fio and fio.contracts regression tests | Writing automated tests for backlog fio and fio.contracts features | Full test coverage for all FIO endpoints and actions |
Devnet release testing | Work with devops to update Devnet test harness and manage all Devnet release testing | Timely Devnet testing |
| Manage all manual and automated Mainnet release testing | Mainnet testing
|
Application testing | Maintain automated application harness (or write UI automated tests). | Timely reporting of new bugs. Development and maintenance of automated tests. | CI Pipeline | Integrated fio.test and application testing into CI pipeline Timely reporting of new bugs found by automated tests |
SDK testing | Run SDK unit tests for every release | SDK tests run within 5 days of SDK releases or within 5 days of fio and fio.contract releases |
Structure
...
Release management |
|
Documentation and knowledge sharing |
|
FIO Chain and contract security | |
EOSIO compatibility and community support | |
Developer outreach and community building |
Centralized
1 x QA Manager
Test harness architecture
Oversee test development
2.5 FTE
fio.test new and backlog automated tests
Proof’d tests
0.5 FTE release testing
Community / Bounty
1 x QA Manager
Test harness architecture
Oversee test development
Create bounties for:
fio.test new and backlog automated tests
0.5 FTE
release testing
Develop/Maintain Proof’d
Outsource
1 x QA Manager
Test harness architecture
Oversee test development with outsource group
Release testing
Outsource
fio.test new and backlog automated tests
Develop/Maintain Proof’d test scripts
...