|
|
# ESITTELY
|
|
# Päätestaussuunnitelma (Master Test Plan)
|
|
|
|
|
|
|
|
|
_Versio 0.9.9.2042018_
|
|
|
|
|
|
|
|
This is the master test plan for a product called Contriboard.
|
|
_Halinallet_
|
|
|
|
|
|
|
|
## Project and project objective
|
|
# Yleistä
|
|
|
Objective of the master test plan
|
|
|
|
|
The objective of the Master Test Plan is to inform all of the individuals involved in testing about the testing process approach, including the mutual relations and activities. This plan describes approach and activities.
|
|
|
|
|
|
|
|
|
|
## Involved in creating the master test plan
|
|
Dokumentissa käydään läpi, mitä työkaluja käytetään Obsimo-palvelun testaamisessa, testausaikataulu, testausstrategia ja testauksen tavoitteet.
|
|
|
This test plan is created by Mikko Soininen and Rikhard Wacker.
|
|
|
|
|
|
|
|
|
|
### Client
|
|
## Testikohteesta (Test Target / System Under Test)
|
|
|
Clients are normal people who are using Contriboard.
|
|
|
|
|
|
|
|
|
|
### Provider
|
|
AWS:n tekniset tiedot tänne.
|
|
|
JAMK University of Applied Sciences provides Challenge Factory with workspace, skills, computers and technology.
|
|
OBSIMO-palvelu, siihen kuuluvat yhteydet ja muut rajapinnat.
|
|
|
|
|
|
|
|
### Scope
|
|
## Testauksen yleiset tavoitteet ja tärkeimmät tehtävät (Test goals and primary needs)
|
|
|
New upcoming testable features can be found in Roadmap. Application's testable functionalites are determined via use cases. Each use case is tested with correct and incorrect inputs for high coverage. Stability, scalability and performance are also in the scope range.
|
|
|
|
|
|
|
|
|
|
### Out of scope
|
|
Testituloksien pitää vastata vaatimusmäärittelyissä määriteltyjä vaatimuksia.
|
|
|
### Acceptants and acceptance criteria
|
|
Tärkeimmät vaatimukset mitkä pitää läpäistä testit ovat:
|
|
|
Acceptance of tested functionality is based on the correct way of working for that specific functionality. Any function cannot overthrow the application. Serious errors can't exist in the compulsory functionalities of the application. Manual test cases are predetermined for passing or failing.
|
|
|
|
|
|
|
|
|
|
# DOCUMENTATION
|
|
- [Käytettävyys](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#k%C3%A4ytett%C3%A4vyys)
|
|
|
|
|
|
|
|
# TEST STRATEGY
|
|
- [Suorituskyky](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#suorituskyky)
|
|
|
Main functionalities are tested via Robot Framework, FMBT and manual testcases.
|
|
|
|
|
|
|
|
|
|
# APPROACH
|
|
- [Tietoturva](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#tietoturva)
|
|
|
Each test scenarios are created by following imaginary stories about clients using the application. For effective automation testing the stories follow an common storyline for a normal user.
|
|
Toiminnallisista vaatimuksista testien pitää läpäistä yli 90%, mikä takaa palvelun toimimisen asiakkaan haluamalla tavalla.
|
|
|
|
|
|
|
|
# TEST LEVELS
|
|
## Aikataulu (Schedule)
|
|
|
### 1. Unit testing
|
|
|
|
|
* The programmer creates functional tests during the coding phase with Karma. Created test can be found here(functional testing).
|
|
|
|
|
|
|
|
|
|
### 2. System testing
|
|
[Linkki projektisuunnitelmaan](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Projektisuunnitelma)
|
|
|
### Functional testing
|
|
|
|
|
|
|
|
|
|
* Automated Robot Framework scenarios are performed in "SUT(System under test)" to find any new failures before integration(regressio).
|
|
## Julkaisusuunnitelma (Release Plan)
|
|
|
|
|
|
|
|
* Manual test case pool is also created, executed and can be found here. "Contriboard" is the selected test project.
|
|
[Linkki alustavaan julkaisusuunnitelmaan](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#julkaisun-suunnitelma)
|
|
|
|
|
|
|
|
* Explorative testing is done during the development phase by programmers.
|
|
## Testausympäristöt (Testing Environments)
|
|
|
|
|
|
|
|
* Regressio testing methods are fMBT. Model based testing formula can be found here
|
|
- Robot Framework
|
|
|
|
|
|
|
|
### Non-functional testing
|
|
- Selenium2
|
|
|
|
|
|
|
|
* Stability will be tested with looped automation test running Robot Framework scenario tests. These tests are run in local client. All the stability test stats are saved for later inspecting and analyzing. Reliability and functionality is tested in addition.
|
|
- Manuaaliset testaukset suoritetaan omalla työpisteellä [Testlinkkiin](http://10.112.7.54/testlink/linkto.php?tprojectPrefix=htc&item=testsuite&id=1399)
|
|
|
|
### Resurssit ja vastuut (Resources and Responsibilities)
|
|
|
|
|
|
|
|
* Locust is used to test performance. Application is stressed with users and then tested how much the system can handle. This testing is done in Foreman test environment for realistic results.
|
|
| Resurssi | Vastuus | Muuta |
|
|
|
|
|:--------:|:-------:|:-----:|
|
|
|
|
| Joona | Suorituskykytestaus | |
|
|
|
|
| Joona | Tietoturvatestas | |
|
|
|
|
| Joona | Toiminnnallinen testaus | |
|
|
|
|
| Joona | Kokeilevat testaus | |
|
|
|
|
| Joona | Käytettävyystestus | |
|
|
|
|
## Testaustasot (Testing Levels)
|
|
|
|
|
|
|
|
### Security testing
|
|
1. Komponenttitason testit
|
|
|
|
|
|
|
|
* Security
|
|
1. Järjestelmän integraatiotestit
|
|
|
|
|
|
|
|
## 3. Acceptance testing
|
|
1. Järjestelmätestit
|
|
|
* Collaborative Testing Sessions are used to validate product quality before moving it in production mode.
|
|
|
|
|
* Automated Robot Framework scenarios are again performed in SUT(System under test) or in "Foreman" environments to find any new failures before acceptance(regressio). This is also the smoke test phase.
|
|
|
|
|
|
|
|
|
|
# TEST ENVIRONMENT
|
|
1. Hyväksyntätestit
|
|
|
## Testing tools
|
|
Eli järjestelmää lähdetään testaamaan pienistä osista kohti isompaa kokonaisuutta, jonka avulla voidaan varmistua järjestelmän toimivuudesta hyväksyntätesteissä.
|
|
|
Same scenarios are used for stability tests...
|
|
|
|
|
|
|
|
|
|
Tools used in testing process:
|
|
### Hyväksyntätestaus (Acceptance Testing)
|
|
|
|
|
|
|
|
* Ubuntu 14.04
|
|
Hyväksyntätestaus suoritetaan automatisoidun Robot Frameworkin ja manuaalitestien avulla.
|
|
|
* Robot Framework
|
|
|
|
|
* RIDE (Robot Framework IDE)
|
|
|
|
|
* Selenium2 library
|
|
|
|
|
* Virtualbox + Vagrant (setting up services for testing environment)
|
|
|
|
|
* fMBT
|
|
|
|
|
* Jenkins
|
|
|
|
|
* Locust
|
|
|
|
|
* Grafana
|
|
|
|
|
* Jarmo
|
|
|
|
|
* Hardware
|
|
|
|
|
* Workstations for local testing
|
|
|
|
|
|
|
|
|
|
# MANAGEMENT
|
|
### Järjestelmätestaus (System Testing)
|
|
|
## Test process
|
|
|
|
|
Testing process for different tests:
|
|
|
|
|
|
|
|
|
|
* Manual test cases are executed in TestLink. Tester follows the test case steps and executes the test in SUT.
|
|
Järjestelmätestaus suoritettaisiin erilaisilla suorituskykyä testaavilla ohjelmilla. Koska suurin osa maksullisia ja palvelu on kuitenkin vielä prototyyppi, niin järjestelmätestaus on tällä hetkellä minimaalinen.
|
|
|
|
|
|
|
|
* Testlink is used as an documentation tool for all the tests, including manual tests. Manual tests are executed and reported in Testlink for later inspection. This gives us the control to observe and manage result statistics during the testing phase.
|
|
### Järjestelmän integraatiotestaus (System Integration Testing)
|
|
|
|
|
|
|
|
* Robot Framework with selenium2 library is used for automation tests. Scenarios are created in Robot Framework and ran with different versions of the application.
|
|
Toteutetaan hyväksyntätestastausken yhteydessä.
|
|
|
|
|
|
|
|
* Stability Tests are run on a local client and executed for long period of times. Stats and version tags are connected for later analyzing.
|
|
### Moduli/komponenttitason testaus (Module / Component Testing)
|
|
|
|
|
|
|
|
* NoiseGenerator is used to create realistic environment for the application. NoiseGenerator creates network delay.
|
|
Toteutetaan Selenium2 ja Robot Frameworkia hyödyntäen.
|
|
|
|
|
|
|
|
* Performance is tested with Foreman and locust.
|
|
## Testaus ja vianhallinnan prosessit
|
|
|
|
|
|
|
|
### Test infrastructure management
|
|
Yleiset hyväksyntäkriteerit
|
|
|
* Test product environment called "Foreman" is created and split on a multiple computers to reduce the load. This test environment is used to run tests during the development continuum. This environment is created for performance testing purpose, it simulates the actual product environment. Build instructions can be found here.
|
|
|
|
|
|
|
|
|
|
* We also have an additional test product environment called "SUT(System under test)". This environment runs on one computer, and it cannot simulate the actual product. Functional and stablility tests can be run on this platform. Multiple users can join this platform. Build process can be found here.
|
|
- Läpäisee testit kriteerien mukaan
|
|
|
|
|
|
|
|
* Robot Framework with selenium2 library is used for automation tests. Only one user can be on the board because this test environment is runned locally. Test client is installed after cloning the github repository. README file tells everything you need to know.
|
|
- Vaatimukset eri ominaisuuksille niiden dokumentaatiossa
|
|
|
|
Yleiset hylkäyskriteerit
|
|
|
|
|
|
|
|
### Test product management
|
|
- Testi ei täytä vaatimuksia
|
|
|
Test environment version is selected by using GitHub tags for version controlling. These can also be found in contriboard's profile -> "about"-dialog. This helps us to determine what version we are testing.
|
|
|
|
|
|
|
|
|
|
Tags can be found under the "branch:master"-button:
|
|
- Testi ei mene hyväksytysti läpi
|
|
|
|
## Valittu testausstrategia
|
|
|
|
|
|
|
|
|
Otetaan ensin huomioon järjestelmälle laaditut vaatimukset. Ominaisuuksia lähdetään testaamaan siinä järjestyksessä, kuin niitä saadaan palveluun tuotettua. Ominaisuuksien luominen palveluun tapahtuu tärkeysjärjestyksessä, mikä samalla priorisoi testausjärjestyksen.
|
|
|
|
|
|
|
|
|
Aluksi on tarkoitus testata toteutettujen ominaisuuksien toimivuutta ja niiden kautta korjailla mahdollisia virheitä. Kun ominaisuuden kaikki testit on suoritettu hyväksytysti, voi testauksessa siirtyä suorituskyky-, kuormitus- ja tietoturvatestauksiin. Näiden jälkeen suoritetaan vielä hyväksyntätestauksia, joiden avulla arvioidaan palvelun valmiutta ja toimivuutta.
|
|
|
|
|
|
|
|
* Client
|
|
## Testityövälineet ja ohjelmistot
|
|
|
* API
|
|
|
|
|
* IMG
|
|
|
|
|
* IO
|
|
|
|
|
* Defects procedure / Error management
|
|
|
|
|
* All defects and enhancement proposals should be reported using GitHub issue tracking, to address and debug defects. Defects and enhancement proposals are classified by different labels. In addition customer can report defects or propose enhancements using UserVoice feature in the product.
|
|
|
|
|
|
|
|
|
|
### Milestones
|
|
### Suorituskykytestaus
|
|
|
Challenge factory is using scrum. Every sprint is a milestone for testing.
|
|
|
|
|
|
|
|
|
|
## Goal
|
|
- https://www.sitespeed.io/
|
|
|
Our main goal is a successful test management and documentation over the project development phase.
|
|
### Toiminnallinen testaus
|
|
|
|
|
|
|
|
## Results
|
|
- Selenium2
|
|
|
* fMBT
|
|
|
|
|
|
|
|
|
|
* Robot framework regressio
|
|
- Robot framework
|
|
|
|
### Tietoturvatestaus
|
|
|
|
|
|
|
|
* Performance Locust
|
|
- Nmap
|
|
|
|
### Kuormitustestaus
|
|
|
|
|
|
|
|
* Manual test case
|
|
- https://jmeter.apache.org/
|
|
|
|
## Liitteet
|
|
|
|
|
|
|
|
* Stability
|
|
- Minttu: Käytettävyys: https://gitlab.labranet.jamk.fi/K8517/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
* Stability with noise
|
|
- Konsta: Stabiilisuus: https://gitlab.labranet.jamk.fi/K8721/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
|
- Jenna: Skaalautuvuus: https://gitlab.labranet.jamk.fi/K2224/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
|
- Joona: Suorituskyky: https://gitlab.labranet.jamk.fi/K1647/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
- Joonas: Ylläpidettävyys https://gitlab.labranet.jamk.fi/K9371/students-private-project/blob/master/PT7/vastaus.md |
|
|
|
\ No newline at end of file |
