Skip to content

6.2 Test Plan

Test execution

Execution of tests in SiVa is divided in two categories: Automated and Manual.

Execution of Automated type of tests

Execution of Unit Tests

These tests are run automatically by Maven every time SiVa project is built. These tests must pass for the build to be successful. It's also possible to execute the tests separately with Maven wrapper using the command:

./mvnw test

These tests are part of SiVa project and can also be executed using IntellJ IDEA. To do that, SiVa project must be loaded into IDEA. Tests are located in test folder in specific SiVa module. To run the tests in IDEA right click on package name and select "Run Tests in <package>". It's also possible to run individual tests by right-clicking on specific test code and selecting "Run <tests name>()".

Execution of System Tests

System tests and instruction for using then can be found in Siva-test repository.

Execution of Manual type of tests

Most of the manual tests require that SiVa service is set up together with SiVa Demo Application. The instructions how to set up SiVa are given in System deployment guide

Execution of manual tests depends on testable area. These tests can be divided into following categories:

  • Statistics tests - test data can be found in StatisticsToLogsSpec.groovy. It can be generated by uploading files into SiVa Demo application. Results have to be verified in logs.
  • Configuration tests - SiVa configuration files have to be modified by hand and service must be set up. Correct behavior of the service must be checked.
  • Other tests - tests are executed by loading files into SiVa Demo application and validating the results shown in the SiVa Demo application.

Files to use in manual tests can be found in SiVa GitHub

System Test introduction

System Tests are carried out automatically on SiVa web service and manually on SiVa Demo application. SiVa web service testing is done using IO RestAssured library to implement automatic checks for REST based tests.

The testing of the SiVa web service is divided into sections based on the software architecture and functionalities provided to the users. The sections are:

  • REST API
  • DDOC container signature validation
  • BDOC container signature validation
  • ASICE container signature validation
  • ASIC-S container signature validation
  • PDF signature validation
  • XAdES hashcode validation

The goal is to focus testing on functionalities implemented in SiVa web service application. Functionalities provided by Validation libraries are not explicitly tested.

In addition, SiVa Demo Application is tested. These tests are carried out manually.

Testing of REST API

The goal of the REST API testing is to check that the API is accepting the requests based on the specification and the output result is in correct format and has all the required elements.

Validation request tests

Following areas are tested on input:

  • Wrong (not accepted) values in input parameters
  • Empty values in input parameters
  • Too many parameters
  • Too few parameters
  • Inconsistencies on stated parameters and actual data (wrong document type)
  • Case insensitivity on parameter names
  • Empty request
  • Request body size limit

In all the negative cases correctness of returned error message is checked.

Get Data Files request tests

Following areas are tested on input:

  • Empty request
  • Empty values in input parameters
  • Too many parameters
  • Too few parameters
  • Changed order of parameters
  • Case insensitivity on parameter names
  • Inconsistencies on stated parameters and actual data
  • Request body size limit

In all the negative cases correctness of returned error message is checked.

Validation report and report signature tests

SiVa web service returns uniform Validation Report on all the supported document types. This also includes correct document types without actual signature (for example PDF document without signature).

Following areas are tested on output (Validation Report):

  • JSON structure on DDOC, BDOC, PDF, ASIC-E, ASIC-S, XAdES hashcode
  • Presence of the mandatory elements on DDOC, BDOC, PDF, ASIC-E, ASIC-S, XAdES hashcode
  • Presence of optional elements on DDOC, BDOC, PDF, ASIC-E, ASIC-S, XAdES hashcode
  • Verification of expected values
  • JSON structure on containers without signatures

Testing of DDOC container signature validation

The goal of the DDOC container signature validation testing is to check that the validation results given by JDigiDoc library are properly presented in validation report.

The testing of DDOC signatures consists of following main cases:

  • Containers with valid signature(s) are validated.
  • Containers with invalid signature(s) or no signature are validated
  • Containers with sizes up to 9MB are validated
  • Containers with DDOC v1.0 - 1.3 are validated

What is not tested:

  • Verification of different causes in container for invalid result is out of scope.

Testing of BDOC container signature validation

The goal of the BDOC container signature validation testing is to check that the validation results given by DigiDoc4J library are properly presented in validation report.

The testing of BDOC container signatures consists of following main cases:

  • Containers with valid signature(s) are validated
  • Containers with invalid signature(s) or no signature are validated
  • Containers with sizes up to 9MB are validated
  • Containers with baseline B, T, LT, LT_TM and LTA profile (files with BDOC extension)

What is not tested:

  • Verification of different causes in container for invalid result is out of scope.

Testing of ASIC-E container signature validation

The goal of the ASIC-E container signature validation testing is to check that the validation results given by DSS library are properly presented in validation report.

The testing of ASIC-E container signatures consists of following main cases:

  • Containers with valid signature(s) are validated
  • Containers with invalid signature(s) or no signature are validated
  • Containers with baseline B, T, LT and LTA profile

What is not tested:

  • Verification of different causes in container for invalid result is out of scope.

Testing of ASIC-S container signature validation

The goal of the ASIC-S container signature validation testing is to check that the validation results given by DSS library are properly presented in validation report.

The testing of ASIC-S container signatures consists of following main cases:

  • Containers with valid signature(s) are validated
  • Containers with invalid signature(s) or no signature are validated
  • Containers with sizes up to 9MB are validated

What is not tested:

  • Verification of different causes in container for invalid result is out of scope.

Testing of XAdES hashcode signature validation

The goal of the XAdES hashcode signature validation testing is to check that the validation results given by DSS library are properly presented in validation report.

The testing of XAdES hashcode signatures consists of following main cases:

  • XAdES signatures with valid signature(s) are validated
  • XAdES signatures with invalid signature(s) or no signature are validated
  • XAdES signatures extracted from BDOC and ASIC-E
  • XAdES signatures with baseline B, T, LT and LTA profile

What is not tested:

  • Verification of different causes in container for invalid result is out of scope.

Testing of PDF signature validation

Portion of the validation rules for PDF documents are implemented in SiVa web application itself. Therefor different test area selection is used for PDF compared to other containers.

The testing of PDF signatures consists of following main cases:

  • Containers with invalid signature(s) (different reasons for failure) are validated
  • Containers with no signature are validated
  • Containers with sizes up to 9MB are validated
  • Containers with different baseline profiles are validated
  • Containers with serial and parallel signatures are validated
  • Containers with different signature cryptographic algorithms are validated
  • Containers with OCSP values inside and outside bounds are validated
  • Containers with baseline B, T, LT and LTA profile

Testing of Data Files Extraction

The goal of the Data Files Extraction testing is to check if right data files returned for DDOC container and error messages for other types of containers.

The testing of Data Files Extraction consists of following main cases:

  • Extracting the data files from valid DDOC container
  • Extracting the data files from Hashcoded DDOC returns null for Base64 encoded String
  • Extracting the data files from DDOC container with 12 different types of files
  • Extracting the data files from not DDOC container
  • Extracting the data files from DDOC container with wrong Document Type

Testing of user statistics

Testing of user statistics is carried out in combination of automatic data preparation and generation by system tests and manual verification of the results. SiVa supports the following way of gathering user statistics:

  • Validation results are printed to system log and can be gathered by any suitable means

Following areas are covered:

  • Statistics values are checked in log for all container types (this also includes parameters not present in validation report)
  • Valid and invalid signatures are validated
  • Error situations on signature validation (instead of validation report, error message is returned)

SiVa Demo Application tests

Testing of SiVa Demo Application is done manually. The main cases are:

  • Cross browser usage (IE, Edge, Chrome, Firefox and Safari)
  • File upload (different sizes, supported and unsupported file types)
  • Displayment of Validation Report for REST
  • Layout of the page
  • Error representation

Configuration/administration testing

Following areas are covered:

  • SiVa Web Application configuration
  • Demo application configuration

Performance testing

More information can be found in SiVa-perftests GitHub page.