QA Strategy
Overview
There are several areas of focus for QA. The following lists those areas including the targeted, actual, and gap resources. Risks to the FIO Protocol are noted.
Project | Notes | Target QA | Actual QA | Gap | Risks |
|---|---|---|---|---|---|
fio and fio.contracts automated testing | Target a 1:2 ratio of QA:Dev | 1.5 | 1 | 0.5 | QA is lagging development when writing QA automated tests which impacts the release. Impact to schedule goes from 2-6 weeks. |
Devnet/Testnet/Mainnet release testing | Involves running regression tests across the various test platforms, logging bugs, and following up on issues | 0.2 | 0 | 0.2 | Eric does all of the release testing. This is sub-optimal and results lighter testing resulting in more bugs getting through and increasing the risk to the chain experiencing unusable features or chain down time. |
FE Projects - Manual testing
| Manual QA testing of dashboard. | 0.3 | 0 | 0.3 | Pawel, GP, and Eric do manual testing. Very little regression testing is done. This is sub-optimal and results lighter testing resulting in more bugs getting through and increasing the risk to the dashboard being offline. Estimated impact to up time: 1-2% |
FE Projects - Automated testing
| Involves either outsourcing of automated test writing or the use of automated test packages. | 0.2 - If we use something like Proof’d automated test software. 1.0 - If we write our own automated tests | 0 | 0.2 - 1.0 | No automated regression tests for the dashboard increasing the risk to bugs getting through and causing the dashboard to be offline. Estimated impact to up time: 2-5% |
|
|
|
|
|
|
Total |
| 2.2 - 3.0 | 1 | 1.2 - 2.0 |
|
* A QA outsourcing group quoted $40K over 3-5 months to do manual testing of the Dashboard and write the initial automated tests. There would also be an ongoing maintenance cost.
** Estimated cost of outsourcing the registration site is approximately 1.5 FTE (~ $5-$10K depending on location). This includes project management, development, and QA. Does not include adding new profiles or working with integrating entities.
QA Models
| FIO Resources | External Resources | Pros | Cons |
|---|---|---|---|---|
Centralized |
|
|
|
|
Community/Bounty |
|
|
|
|
Outsourced |
|
|
|
|
Discussion
Several factors can be considered when analyzing the models above:
Metric | Current | Analysis |
|---|---|---|
Budget | FIO currently operates as a startup, has a limited budget, and is not in a high growth mode. | Outsourcing full projects is generally more expensive than retaining in house because of the additional overhead built into the costs. Recommendation: Centralized or Community/Bounty |
Project Scope | With the exception of Dashboard, FIO does not have many multi-month multi-FTE projects. If you combined projects like staking and wrapping, you could possibly count as large projects, albeit with limited duration needed for QA (4-6 weeks) | Outsourcing is better suited to larger projects with longer durations because of the overhead that will need to be taken on to train QA engineers on the FIO Protocol. It would be difficult to convince a group to take on a 0.5 FTE 2 month project. Generally you want to target at least 2-4 FTE to fully outsource. Recommendation: Centralized or Community/Bounty |
FIO Structure | FIO currently operates as a centralized organization but would like to move to more of a DAO model where the community is heavily engaged with development. | If the goal is to engage with a wider range of developers across the ecosystem, the outsourcing model may make more sense. Building an internal team can create an insider/outsider view of FIO development and discourage external development resources from engaging with FIO Recommendation: Community/Bounty or Outsourced |
Eric recommendation
Given the current state of FIO (limited budget and a focus on smaller projects) I think it makes sense to continue to build the internal team but drive towards outsourcing the tasks that require less expertise via bounty or pure outsourced programs. A cost analysis would have to be done to determine which of these models makes more financial sense and which makes more “community” sense. There is a tradeoff. Once FIO hits a medium to high growth curve it would make sense to revisit this model and begin to outsource the larger projects. I would also recommend we step into outsourcing by encouraging AlephOne to take on a larger project management and QA role in the development of the dashboard.
Short term: Focus on Community/Bounty model where a strong, expert internal team is developed but project that require less expertise are outsourced as bounties or fully outsourced.
Medium term: Encourage more self-reliant outsourcing for larger projects, such as dashboard.
Long term: As FIO moves into a higher-growth model build broader outsourcing teams and infrastructure.