Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Current »

Overview

There are several areas of focus for QA. The following lists those areas including the targeted, actual, and gap resources. Risks to the FIO Protocol are noted.

Project

Notes

Target QA

Actual QA

Gap

Risks

fio and fio.contracts automated testing

Target a 1:2 ratio of QA:Dev

1.5

1

0.5

QA is lagging development when writing QA automated tests which impacts the release.

Impact to schedule goes from 2-6 weeks.

Devnet/Testnet/Mainnet release testing

Involves running regression tests across the various test platforms, logging bugs, and following up on issues

0.2

0

0.2

Eric does all of the release testing. This is sub-optimal and results lighter testing resulting in more bugs getting through and increasing the risk to the chain experiencing unusable features or chain down time.

FE Projects - Manual testing

  • Dashboard*

  • Registration Site

Manual QA testing of dashboard.

0.3

0

0.3

Pawel, GP, and Eric do manual testing. Very little regression testing is done. This is sub-optimal and results lighter testing resulting in more bugs getting through and increasing the risk to the dashboard being offline.

Estimated impact to up time: 1-2%

FE Projects - Automated testing

  • Dashboard*

  • Registration Site

Involves either outsourcing of automated test writing or the use of automated test packages.

0.2 - If we use something like Proof’d automated test software.

1.0 - If we write our own automated tests

0

0.2 - 1.0

No automated regression tests for the dashboard increasing the risk to bugs getting through and causing the dashboard to be offline.

Estimated impact to up time: 2-5%

Total

2.2 - 3.0

1

1.2 - 2.0

* A QA outsourcing group quoted $40K over 3-5 months to do manual testing of the Dashboard and write the initial automated tests. There would also be an ongoing maintenance cost.

** Estimated cost of outsourcing the registration site is approximately 1.5 FTE (~ $5-$10K depending on location). This includes project management, development, and QA. Does not include adding new profiles or working with integrating entities.

QA Models

FIO Resources

External Resources

Pros

Cons

Centralized

  • 1 x QA Manager

    • Test harness architecture

    • Oversee test development

  • 1.5 FTE

    • Based on current chain, FIP, and projects

    • fio.test new and backlog automated tests

  • Expertise - Enables development of expertise

  • Flexibility - Allows for distribution of QA resources across smaller multiple projects

  • Scalability - Does not scale with larger projects

Community/Bounty

  • 1 x QA Manager

    • Test harness architecture

    • Oversee test development

    • Release testing

  • Create bounties for:

    • fio.test new and backlog automated tests

  • 0.5 FTE

    • Release testing

    • Develop/Maintain Proof’d

  • 1 x FTE

    • fio.test new and backlog automated tests

  • Expertise - Retains upper level experienced QA

  • Community - Leverages the community for tasks that require less expertise

  • Scalability - Does not scale with larger projects

Outsourced

  • 1 x QA Manager

    • Test harness architecture

    • Oversee test development with outsource group

    • Release testing

  • 2 x FTE

    • fio.test new and backlog automated tests

    • Develop/Maintain Proof’d test scripts

  • Scalability - Scales well for larger projects

  • Overhead - Requires overhead of managing external projects

Discussion

Several factors can be considered when analyzing the models above:

Metric

Current

Analysis

Budget

FIO currently operates as a startup, has a limited budget, and is not in a high growth mode.

Outsourcing full projects is generally more expensive than retaining in house because of the additional overhead built into the costs.

Recommendation: Centralized or Community/Bounty

Project Scope

With the exception of Dashboard, FIO does not have many multi-month multi-FTE projects. If you combined projects like staking and wrapping, you could possibly count as large projects, albeit with limited duration needed for QA (4-6 weeks)

Outsourcing is better suited to larger projects with longer durations because of the overhead that will need to be taken on to train QA engineers on the FIO Protocol. It would be difficult to convince a group to take on a 0.5 FTE 2 month project. Generally you want to target at least 2-4 FTE to fully outsource.

Recommendation: Centralized or Community/Bounty

FIO Structure

FIO currently operates as a centralized organization but would like to move to more of a DAO model where the community is heavily engaged with development.

If the goal is to engage with a wider range of developers across the ecosystem, the outsourcing model may make more sense. Building an internal team can create an insider/outsider view of FIO development and discourage external development resources from engaging with FIO

Recommendation: Community/Bounty or Outsourced

Eric recommendation

Given the current state of FIO (limited budget and a focus on smaller projects) I think it makes sense to continue to build the internal team but drive towards outsourcing the tasks that require less expertise via bounty or pure outsourced programs. A cost analysis would have to be done to determine which of these models makes more financial sense and which makes more “community” sense. There is a tradeoff. Once FIO hits a medium to high growth curve it would make sense to revisit this model and begin to outsource the larger projects. I would also recommend we step into outsourcing by encouraging AlephOne to take on a larger project management and QA role in the development of the dashboard.

  • No labels