|
|
|
# ESITTELY
|
|
|
|
# Päätestaussuunnitelma (Master Test Plan)
|
|
|
|
|
|
|
|
_Versio 0.9.9.2042018_
|
|
|
|
|
|
|
|
This is the master test plan for a product called Contriboard.
|
|
|
|
_Halinallet_
|
|
|
|
|
|
|
|
## Project and project objective
|
|
|
|
Objective of the master test plan
|
|
|
|
The objective of the Master Test Plan is to inform all of the individuals involved in testing about the testing process approach, including the mutual relations and activities. This plan describes approach and activities.
|
|
|
|
# Yleistä
|
|
|
|
|
|
|
|
## Involved in creating the master test plan
|
|
|
|
This test plan is created by Mikko Soininen and Rikhard Wacker.
|
|
|
|
Dokumentissa käydään läpi, mitä työkaluja käytetään Obsimo-palvelun testaamisessa, testausaikataulu, testausstrategia ja testauksen tavoitteet.
|
|
|
|
|
|
|
|
### Client
|
|
|
|
Clients are normal people who are using Contriboard.
|
|
|
|
## Testikohteesta (Test Target / System Under Test)
|
|
|
|
|
|
|
|
### Provider
|
|
|
|
JAMK University of Applied Sciences provides Challenge Factory with workspace, skills, computers and technology.
|
|
|
|
AWS:n tekniset tiedot tänne.
|
|
|
|
OBSIMO-palvelu, siihen kuuluvat yhteydet ja muut rajapinnat.
|
|
|
|
|
|
|
|
### Scope
|
|
|
|
New upcoming testable features can be found in Roadmap. Application's testable functionalites are determined via use cases. Each use case is tested with correct and incorrect inputs for high coverage. Stability, scalability and performance are also in the scope range.
|
|
|
|
## Testauksen yleiset tavoitteet ja tärkeimmät tehtävät (Test goals and primary needs)
|
|
|
|
|
|
|
|
### Out of scope
|
|
|
|
### Acceptants and acceptance criteria
|
|
|
|
Acceptance of tested functionality is based on the correct way of working for that specific functionality. Any function cannot overthrow the application. Serious errors can't exist in the compulsory functionalities of the application. Manual test cases are predetermined for passing or failing.
|
|
|
|
Testituloksien pitää vastata vaatimusmäärittelyissä määriteltyjä vaatimuksia.
|
|
|
|
Tärkeimmät vaatimukset mitkä pitää läpäistä testit ovat:
|
|
|
|
|
|
|
|
# DOCUMENTATION
|
|
|
|
- [Käytettävyys](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#k%C3%A4ytett%C3%A4vyys)
|
|
|
|
|
|
|
|
# TEST STRATEGY
|
|
|
|
Main functionalities are tested via Robot Framework, FMBT and manual testcases.
|
|
|
|
- [Suorituskyky](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#suorituskyky)
|
|
|
|
|
|
|
|
# APPROACH
|
|
|
|
Each test scenarios are created by following imaginary stories about clients using the application. For effective automation testing the stories follow an common storyline for a normal user.
|
|
|
|
- [Tietoturva](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#tietoturva)
|
|
|
|
Toiminnallisista vaatimuksista testien pitää läpäistä yli 90%, mikä takaa palvelun toimimisen asiakkaan haluamalla tavalla.
|
|
|
|
|
|
|
|
# TEST LEVELS
|
|
|
|
### 1. Unit testing
|
|
|
|
* The programmer creates functional tests during the coding phase with Karma. Created test can be found here(functional testing).
|
|
|
|
## Aikataulu (Schedule)
|
|
|
|
|
|
|
|
### 2. System testing
|
|
|
|
### Functional testing
|
|
|
|
[Linkki projektisuunnitelmaan](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Projektisuunnitelma)
|
|
|
|
|
|
|
|
* Automated Robot Framework scenarios are performed in "SUT(System under test)" to find any new failures before integration(regressio).
|
|
|
|
## Julkaisusuunnitelma (Release Plan)
|
|
|
|
|
|
|
|
* Manual test case pool is also created, executed and can be found here. "Contriboard" is the selected test project.
|
|
|
|
[Linkki alustavaan julkaisusuunnitelmaan](https://gitlab.labranet.jamk.fi/PRJTEAM-H/halinallet/wikis/Vaatimusmaaritelma#julkaisun-suunnitelma)
|
|
|
|
|
|
|
|
* Explorative testing is done during the development phase by programmers.
|
|
|
|
## Testausympäristöt (Testing Environments)
|
|
|
|
|
|
|
|
* Regressio testing methods are fMBT. Model based testing formula can be found here
|
|
|
|
- Robot Framework
|
|
|
|
|
|
|
|
### Non-functional testing
|
|
|
|
- Selenium2
|
|
|
|
|
|
|
|
* Stability will be tested with looped automation test running Robot Framework scenario tests. These tests are run in local client. All the stability test stats are saved for later inspecting and analyzing. Reliability and functionality is tested in addition.
|
|
|
|
- Manuaaliset testaukset suoritetaan omalla työpisteellä [Testlinkkiin](http://10.112.7.54/testlink/linkto.php?tprojectPrefix=htc&item=testsuite&id=1399)
|
|
|
|
### Resurssit ja vastuut (Resources and Responsibilities)
|
|
|
|
|
|
|
|
* Locust is used to test performance. Application is stressed with users and then tested how much the system can handle. This testing is done in Foreman test environment for realistic results.
|
|
|
|
| Resurssi | Vastuus | Muuta |
|
|
|
|
|:--------:|:-------:|:-----:|
|
|
|
|
| Joona | Suorituskykytestaus | |
|
|
|
|
| Joona | Tietoturvatestas | |
|
|
|
|
| Joona | Toiminnnallinen testaus | |
|
|
|
|
| Joona | Kokeilevat testaus | |
|
|
|
|
| Joona | Käytettävyystestus | |
|
|
|
|
## Testaustasot (Testing Levels)
|
|
|
|
|
|
|
|
### Security testing
|
|
|
|
1. Komponenttitason testit
|
|
|
|
|
|
|
|
* Security
|
|
|
|
1. Järjestelmän integraatiotestit
|
|
|
|
|
|
|
|
## 3. Acceptance testing
|
|
|
|
* Collaborative Testing Sessions are used to validate product quality before moving it in production mode.
|
|
|
|
* Automated Robot Framework scenarios are again performed in SUT(System under test) or in "Foreman" environments to find any new failures before acceptance(regressio). This is also the smoke test phase.
|
|
|
|
1. Järjestelmätestit
|
|
|
|
|
|
|
|
# TEST ENVIRONMENT
|
|
|
|
## Testing tools
|
|
|
|
Same scenarios are used for stability tests...
|
|
|
|
1. Hyväksyntätestit
|
|
|
|
Eli järjestelmää lähdetään testaamaan pienistä osista kohti isompaa kokonaisuutta, jonka avulla voidaan varmistua järjestelmän toimivuudesta hyväksyntätesteissä.
|
|
|
|
|
|
|
|
Tools used in testing process:
|
|
|
|
### Hyväksyntätestaus (Acceptance Testing)
|
|
|
|
|
|
|
|
* Ubuntu 14.04
|
|
|
|
* Robot Framework
|
|
|
|
* RIDE (Robot Framework IDE)
|
|
|
|
* Selenium2 library
|
|
|
|
* Virtualbox + Vagrant (setting up services for testing environment)
|
|
|
|
* fMBT
|
|
|
|
* Jenkins
|
|
|
|
* Locust
|
|
|
|
* Grafana
|
|
|
|
* Jarmo
|
|
|
|
* Hardware
|
|
|
|
* Workstations for local testing
|
|
|
|
Hyväksyntätestaus suoritetaan automatisoidun Robot Frameworkin ja manuaalitestien avulla.
|
|
|
|
|
|
|
|
# MANAGEMENT
|
|
|
|
## Test process
|
|
|
|
Testing process for different tests:
|
|
|
|
### Järjestelmätestaus (System Testing)
|
|
|
|
|
|
|
|
* Manual test cases are executed in TestLink. Tester follows the test case steps and executes the test in SUT.
|
|
|
|
Järjestelmätestaus suoritettaisiin erilaisilla suorituskykyä testaavilla ohjelmilla. Koska suurin osa maksullisia ja palvelu on kuitenkin vielä prototyyppi, niin järjestelmätestaus on tällä hetkellä minimaalinen.
|
|
|
|
|
|
|
|
* Testlink is used as an documentation tool for all the tests, including manual tests. Manual tests are executed and reported in Testlink for later inspection. This gives us the control to observe and manage result statistics during the testing phase.
|
|
|
|
### Järjestelmän integraatiotestaus (System Integration Testing)
|
|
|
|
|
|
|
|
* Robot Framework with selenium2 library is used for automation tests. Scenarios are created in Robot Framework and ran with different versions of the application.
|
|
|
|
Toteutetaan hyväksyntätestastausken yhteydessä.
|
|
|
|
|
|
|
|
* Stability Tests are run on a local client and executed for long period of times. Stats and version tags are connected for later analyzing.
|
|
|
|
### Moduli/komponenttitason testaus (Module / Component Testing)
|
|
|
|
|
|
|
|
* NoiseGenerator is used to create realistic environment for the application. NoiseGenerator creates network delay.
|
|
|
|
Toteutetaan Selenium2 ja Robot Frameworkia hyödyntäen.
|
|
|
|
|
|
|
|
* Performance is tested with Foreman and locust.
|
|
|
|
## Testaus ja vianhallinnan prosessit
|
|
|
|
|
|
|
|
### Test infrastructure management
|
|
|
|
* Test product environment called "Foreman" is created and split on a multiple computers to reduce the load. This test environment is used to run tests during the development continuum. This environment is created for performance testing purpose, it simulates the actual product environment. Build instructions can be found here.
|
|
|
|
Yleiset hyväksyntäkriteerit
|
|
|
|
|
|
|
|
* We also have an additional test product environment called "SUT(System under test)". This environment runs on one computer, and it cannot simulate the actual product. Functional and stablility tests can be run on this platform. Multiple users can join this platform. Build process can be found here.
|
|
|
|
- Läpäisee testit kriteerien mukaan
|
|
|
|
|
|
|
|
* Robot Framework with selenium2 library is used for automation tests. Only one user can be on the board because this test environment is runned locally. Test client is installed after cloning the github repository. README file tells everything you need to know.
|
|
|
|
- Vaatimukset eri ominaisuuksille niiden dokumentaatiossa
|
|
|
|
Yleiset hylkäyskriteerit
|
|
|
|
|
|
|
|
### Test product management
|
|
|
|
Test environment version is selected by using GitHub tags for version controlling. These can also be found in contriboard's profile -> "about"-dialog. This helps us to determine what version we are testing.
|
|
|
|
- Testi ei täytä vaatimuksia
|
|
|
|
|
|
|
|
Tags can be found under the "branch:master"-button:
|
|
|
|
- Testi ei mene hyväksytysti läpi
|
|
|
|
## Valittu testausstrategia
|
|
|
|
|
|
|
|
Otetaan ensin huomioon järjestelmälle laaditut vaatimukset. Ominaisuuksia lähdetään testaamaan siinä järjestyksessä, kuin niitä saadaan palveluun tuotettua. Ominaisuuksien luominen palveluun tapahtuu tärkeysjärjestyksessä, mikä samalla priorisoi testausjärjestyksen.
|
|
|
|
|
|
|
|
Aluksi on tarkoitus testata toteutettujen ominaisuuksien toimivuutta ja niiden kautta korjailla mahdollisia virheitä. Kun ominaisuuden kaikki testit on suoritettu hyväksytysti, voi testauksessa siirtyä suorituskyky-, kuormitus- ja tietoturvatestauksiin. Näiden jälkeen suoritetaan vielä hyväksyntätestauksia, joiden avulla arvioidaan palvelun valmiutta ja toimivuutta.
|
|
|
|
|
|
|
|
* Client
|
|
|
|
* API
|
|
|
|
* IMG
|
|
|
|
* IO
|
|
|
|
* Defects procedure / Error management
|
|
|
|
* All defects and enhancement proposals should be reported using GitHub issue tracking, to address and debug defects. Defects and enhancement proposals are classified by different labels. In addition customer can report defects or propose enhancements using UserVoice feature in the product.
|
|
|
|
## Testityövälineet ja ohjelmistot
|
|
|
|
|
|
|
|
### Milestones
|
|
|
|
Challenge factory is using scrum. Every sprint is a milestone for testing.
|
|
|
|
### Suorituskykytestaus
|
|
|
|
|
|
|
|
## Goal
|
|
|
|
Our main goal is a successful test management and documentation over the project development phase.
|
|
|
|
- https://www.sitespeed.io/
|
|
|
|
### Toiminnallinen testaus
|
|
|
|
|
|
|
|
## Results
|
|
|
|
* fMBT
|
|
|
|
- Selenium2
|
|
|
|
|
|
|
|
* Robot framework regressio
|
|
|
|
- Robot framework
|
|
|
|
### Tietoturvatestaus
|
|
|
|
|
|
|
|
* Performance Locust
|
|
|
|
- Nmap
|
|
|
|
### Kuormitustestaus
|
|
|
|
|
|
|
|
* Manual test case
|
|
|
|
- https://jmeter.apache.org/
|
|
|
|
## Liitteet
|
|
|
|
|
|
|
|
* Stability
|
|
|
|
- Minttu: Käytettävyys: https://gitlab.labranet.jamk.fi/K8517/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
* Stability with noise
|
|
|
|
- Konsta: Stabiilisuus: https://gitlab.labranet.jamk.fi/K8721/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
- Jenna: Skaalautuvuus: https://gitlab.labranet.jamk.fi/K2224/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
- Joona: Suorituskyky: https://gitlab.labranet.jamk.fi/K1647/students-private-project/blob/master/PT7/vastaus.md
|
|
|
|
|
|
|
|
- Joonas: Ylläpidettävyys https://gitlab.labranet.jamk.fi/K9371/students-private-project/blob/master/PT7/vastaus.md |
|
|
\ No newline at end of file |
