Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Defines the tasks associated with FIO QA. FIO QA is mainly focused on automated test coverage and is less concerned with defining manual test cases that can be followed by a QA engineer.

Overview

There are several categories of FIO QA:

fio and fio.contracts automated testing

fio.test javascript tests

Devnet release testing

Testnet release testing

Mainnet release testing

Application testing

Includes manual and automated testing of FIO applications:

  • Dashboard

  • Registration Site

SDK testing

Running of SDK unit tests for every release.

QA Tasks

fio and fio.contracts automated testing

fio and fio.contract QA is covered by the fio.test javascript test harness. New automated tests are created for every new action, endpoint and feature added to the protocol. Full regression tests are run for every release.

The current process for developing automated tests generally follows:

  • FIP/requirements are defined by product team

  • Product team publishes initial test cases

  • Test cases are enhanced by development and QA teams

  • FIO development creates initial tests as a template

  • FIO QA completes full automated tests

Gaps in testing:

  • All features, especially Mainnet features, do not have full regression tests

  • Testing lags development and impacts releases

  • Limited QA resources to develop automated tests

  • No time to focus on test harness architecture

  • QA not integrated into CI pipeline

Devnet release testing

Devnet release testing is currently done by the development team. It involves deployment and upgrade testing of fio and fio.contracts.

Testnet release testing

Testnet release testing currently consists of:

  • Running an automated smoketest against Testnet before and after deployment

  • Some manual testing of Testnet using Cryptonym

  • Some manual testing of features by development and product teams

Mainnet release testing

Mainnet release testing currently consists of:

  • Some manual testing of Mainnet using Cryptonym

  • Some manual testing of features by development and product teams

Application testing

Application testing is currently covered by manual UAT testing by the development and product teams. Options being considered are:

  • Using Proof’d automated regression scans to catch changes

  • Writing Web UI automated tests

  • Increasing manual testing

SDK testing

SDK testing currently consists of:

  • SDK unit tests written by development team

  • Typescript unit tests run against new releases of fiosdk_typescript and new fio and fio.contracts releases

  • Kotlin and Swift unit tests are not currently maintained or run for new releases.

  • Go SDK is maintained by Blockpane. TBD on how unit tests are built or run. TBD on how Go SDK is maintained.

Resource estimates

  • 1 FTE for updating test harness and overseeing QA automated test development

  • 1 FTE for adding missing regression tests (can be done by multiple developers)

    • Bounties or outsourcing?

  • 1.5 FTE for adding new regression tests for new features (2:1 ratio of developers to QA)

  • 0.25 FTE to maintain Proof’d or other automated test suite.

  • 0.2 FTE for release testing

  • 0.1 FTE for SDK release testing (does not include SDK unit test development)

Project Plan and KPIs

There are several categories of FIO QA:

Task

Description

KPIs

fio and fio.contracts new feature

Writing automated tests for new fio and fio.contracts features

Timely delivery of automated tests for new features. TBD on this process.

Backlog of fio and fio.contracts regression tests

Writing automated tests for backlog fio and fio.contracts features

Full test coverage for all FIO endpoints and actions

Devnet release testing

Work with devops to update Devnet test harness and manage all Devnet release testing

Timely Devnet testing

Testnet release testing

Manage all manual and automated Testnet release testing

Testnet testing completed within 2 hours of new releases

Mainnet release testing

Manage all manual and automated Mainnet release testing

Mainnet testing completed within 2 hours of new releases

Application testing

Maintain automated application harness (or write UI automated tests).

Timely reporting of new bugs.

Development and maintenance of automated tests.

CI Pipeline

Integrated fio.test and application testing into CI pipeline

Timely reporting of new bugs found by automated tests

SDK testing

Run SDK unit tests for every release

SDK tests run within 5 days of SDK releases or within 5 days of fio and fio.contract releases

Structure

Possible QA org structures

Centralized

  • 1 x QA Manager

    • Test harness architecture

    • Oversee test development

  • 2.5 FTE

    • fio.test new and backlog automated tests

    • Proof’d tests

  • 0.5 FTE release testing

Community / Bounty

  • 1 x QA Manager

    • Test harness architecture

    • Oversee test development

  • Create bounties for:

    • fio.test new and backlog automated tests

  • 0.5 FTE

    • release testing

    • Develop/Maintain Proof’d

Outsource

  • 1 x QA Manager

    • Test harness architecture

    • Oversee test development with outsource group

    • Release testing

  • Outsource

    • fio.test new and backlog automated tests

    • Develop/Maintain Proof’d test scripts

  • No labels