6.2 Test Plan

Test execution

Execution of tests in SiVa is divided in three categories: Automated, Automated SoapUI, Manual. The method is also indicated in test case description with TestType field.

Execution of Automated type of tests

These tests are run automatically by Maven every time SiVa project is built. These tests must pass for the build to be successful. Its also possible to execute the tests separately with Maven wrapper using the command:

./mvnw verify

These tests are part of SiVa project and can also be executed using IntellJ IDEA. To do that, SiVa project must be loaded into IDEA. Tests are located in following packages:

• ee.openeid.siva.integrationtest
• ee.openeid.siva.resttest
• ee.openeid.siva.soaptest

To run the tests in IDEA right click on package name and select "Run tests in". It's also possible to run individual tests by right clicking on specific test code and selecting "Run".

Execution of Automated SoapUI type of tests

The description how to execute the tests together with SoapUI project file to be used can be found in SiVa GitHub

Execution of Manual type of tests

Most of the manual tests require that SiVa service is set up together with SiVa Demo Application. The instructions how to set up SiVa are given in System deployment guide

Execution of manual tests depends on testable area. These tests can be divided into following categories:

• Statistics tests - test data can be prepeared with exectuting ee.openeid.siva.manualtest package. It can be also generated by uploading files into SiVa Demo application. Results have to be verified in logs.
• Report tests - test data can be prepeared with exectuting ee.openeid.siva.manualtest package. Results have to be verified manually.
• Configuration tests - SiVa configuration files have to be modified by hand and service must be set up. Correct behavior of the service must be checked.
• Other tests - tests are executed by loading files into SiVa Demo application and validating the results shown in the SiVa Demo application.

Files to use in manual tests can be found in SiVa GitHub

Integration Test introduction

This section of the document gives overview of Integration Testing carried out on SiVa web service and SiVa Demo application.

SiVa web service Integration Testing is using IO RestAssured library to implement automatic checks for REST/SOAP based tests.

The testing of the SiVa web service is divided into sections based on the software architecture and functionalities provided to the users. The sections are:

• REST API
• SOAP API
• DDOC container signature validation
• BDOC container signature validation
• ASICE container signature validation
• ASIC-S container signature validation
• PDF signature validation
• XAdES hashcode validation

The goal is to focus testing on functionalities implemented in SiVa web service application. Functionalities provided by Validation libraries are not explicitly tested.

In addition SiVa Demo Application is tested. These tests are carried out manually.

Testing of REST API

The goal of the REST API testing is to check that the API is accepting the requests based on the specification and the output result is in correct format and has all the required elements.

Validation request tests

Following areas are tested on input:

• Wrong (not accepted) values in input parameters
• Empty values in input paramters
• Too many parameters
• Too few parameters
• Inconsistencies on stated parameters and actual data (wrong document type)
• Case insensitivity on parameter names
• Empty request
• Request body size limit

In all of the negative cases correctness of returned error message is checked.

Specific test cases and input files can be found in:

• ValidationRequestIT.java
• DocumentFormatIT.java
• HashcodeValidationRequestIT.java

Get Data Files request tests

Following areas are tested on input:

• Empty request
• Empty values in input parameters
• Too many parameters
• Too few parameters
• Changed order of parameters
• Case insensitivity on parameter names
• Inconsistencies on stated parameters and actual data
• Request body size limit

In all of the negative cases correctness of returned error message is checked.

Specific test cases and input files can be found in:

• GetDataFileRequestIT.java

Validation report and report siganture tests

SiVa web service returns uniform Validation Report on all the supported document types. This also includes correct document types without actual signature (for example PDF document without signature).

Following areas are tested on output (Validation Report):

• JSON structure on DDOC, BDOC, PDF, ASIC-E, ASIC-S, XAdES hashcode
• Presence of the mandatory elements on DDOC, BDOC, PDF, ASIC-E, ASIC-S, XAdES hashcode
• Presence of optional elements on DDOC, BDOC, PDF, ASIC-E, ASIC-S, XAdES hashcode
• Verification of expected values
• JSON structure on containers without signatures

Specific test cases and input files can be found in:

• ValidationReportValueVerificationIT.java
• ReportSignatureManualIT.java
• DetailedReportValidationManualIT.java
• DiagnosticReportValidationManualIT.java

Testing of SOAP API

The goal of the SOAP API testing is to check that the API is accepting the requests based on the specification and the output result (Validation Report) is in correct format and has all the required elements. In general the tests follow the same principles as with REST API. Compatibility with X-Road security server is out of scope for these tests and will be covered in X-Road System Test plan.

Validation request tests

Following areas are tested on input:

• Wrong (not accepted) values in input parameters
• Empty values in input paramters
• Too many parameters
• Too few parameters
• Inconsistencies on stated parameters and actual data (wrong document type)
• Case insensitivity on parameter names
• Empty request

In all of the negative cases correctness of returned error message is checked.

Specific test cases and input files can be found in:

• SoapValidationRequestIT.java
• SoapHashcodeValidationRequestIT.java

Get Data Files request tests

Following areas are tested on input:

• Empty request
• Empty values in input parameters
• Too many parameters
• Too few parameters
• Changed order of parameters
• Case insensitivity on parameter names
• Inconsistencies on stated parameters and actual data

In all of the negative cases correctness of returned error message is checked.

Specific test cases and input files can be found in:

• SoapGetDataFileRequestIT.java

Validation report tests

SiVa web service returns uniform Validation Report on all the supported document types. This also includes correct document types without actual signature (for example PDF document without signature). However not all values may be present for all the document types.

Following areas are tested on output (Validation Report):

• Presence of the mandatory elements on DDOC, BDOC, PDF, ASIC-S ASIC-E, XAdES hashcode
• Presence of optional elements on DDOC, BDOC, PDF, ASIC-S, ASIC-E, XAdES hashcode and
• Verification of expected values

Specific test cases and input files can be found in:

• SoapValidationReportValueIT.java

Get Data Files report tests

Following areas are tested on output:

• Presence of the mandatory elements
• Verification of expected values
• Extraction of all data files

Specific test cases and input files can be found in:

• SoapGetDataFileReportIT.java

Testing of DDOC container signature validation

The goal of the DDOC container signature validation testing is to check that the validation results given by JDigiDoc library are properly presented in validation report.

The testing of DDOC signatures consists of following main cases:

• Containers with valid signature(s) are validated.
• Containers with invalid signature(s) or no signature are validated
• Containers with sizes up to 9MB are validated
• Containers with DDOC v1.0 - 1.3 are validated

Specific test cases and input files can be found in:

• DdocValidationFailIT.java
• DdocValidationPassIT.java
• LargeFileIT.java

What is not tested:

• Verification of different causes in container for invalid result is out of scope.

Testing of BDOC container signature validation

The goal of the BDOC container signature validation testing is to check that the validation results given by DigiDoc4J library are properly presented in validation report.

The testing of BDOC container signatures consists of following main cases:

• Containers with valid signature(s) are validated
• Containers with invalid signature(s) or no signature are validated
• Containers with sizes up to 9MB are validated
• Containers with baseline B, T, LT, LT_TM and LTA profile (files with BDOC extension)

Specific test cases and input files can be found in:

• BdocValidationFailIT.java
• BdocValidationPassIT.java
• LargeFileIT.java

What is not tested:

• Verification of different causes in container for invalid result is out of scope.

Testing of ASIC-E container signature validation

The goal of the ASIC-E container signature validation testing is to check that the validation results given by DSS library are properly presented in validation report.

The testing of ASIC-E container signatures consists of following main cases:

• Containers with valid signature(s) are validated
• Containers with invalid signature(s) or no signature are validated
• Containers with baseline B, T, LT and LTA profile

Specific test cases and input files can be found in:

• AsiceValidationFailIT.java
• AsiceValidationPassIT.java

What is not tested:

• Verification of different causes in container for invalid result is out of scope.

Testing of ASIC-S container signature validation

The goal of the ASIC-S container signature validation testing is to check that the validation results given by DSS library are properly presented in validation report.

The testing of ASIC-S container signatures consists of following main cases:

• Containers with valid signature(s) are validated
• Containers with invalid signature(s) or no signature are validated
• Containers with sizes up to 9MB are validated

Specific test cases and input files can be found in:

• AsicsValidationFailIT.java
• AsicsValidationPassIT.java

What is not tested:

• Verification of different causes in container for invalid result is out of scope.

Testing of XAdES hashcode signature validation

The goal of the XAdES hashcode signature validation testing is to check that the validation results given by DSS library are properly presented in validation report.

The testing of XAdES hashcode signatures consists of following main cases:

• XAdES signatures with valid signature(s) are validated
• XAdES signatures with invalid signature(s) or no signature are validated
• XAdES signatures extracted from BDOC and ASIC-E
• XAdES signatures with baseline B, T, LT and LTA profile

Specific test cases and input files can be found in:

• XadesHashcodeValidationFailIT.java
• XadesHashcodeValidationPassIT.java

What is not tested:

• Verification of different causes in container for invalid result is out of scope.

Testing of PDF signature validation

Portion of the validation rules for PDF documents are implemented in SiVa web application itself. Therefor different test area selection is used for PDF compared to other containers.

The testing of PDF signatures consists of following main cases:

• Containers with invalid signature(s) (different reasons for failure) are validated
• Containers with no signature are validated
• Containers with sizes up to 9MB are validated
• Containers with different baseline profiles are validated
• Containers with serial and parallel signatures are validated
• Containers with different signature cryptocaphic algorithms are validated
• Containers with OCSP values inside and outside bounds are validated
• Containers with baseline B, T, LT and LTA profile

Specific test cases and input files can be found in:

• PdfBaselineProfileIT.java
• PdfSignatureCryptographicAlgorithmIT.java
• PdfValidationFailIT.java
• PdfValidationPassIT.java
• LargeFileIT.java

Testing of Data Files Extraction

The goal of the Data Files Extraction testing is to check if right data files returned for DDOC container and error messages for other types of containers.

The testing of Data Files Extraction consists of following main cases:

• Extracting the data files from valid DDOC container
• Extracting the data files from Hashcoded DDOC returns null for Base64 encoded String
• Extracting the data files from DDOC container with 12 different types of files
• Extracting the data files from not DDOC container
• Extracting the data files from DDOC container with wrong Document Type

Specific test cases and input files can be found in:

• DdocGetDataFilesIT.java

Testing of user statistics

Testing of user statistics is carried out in combination of automatic data preparation and generation by integration tests and manual verification of the results. SiVa supports the following way of gathering user statistics:

• Validation results are printed to system log and can be gathered by any suitable means

Following areas are covered:

• Statistics values are checked in log for all container types (this also includes parameters not present in validation report)
• Valid and invalid signatures are validated
• Error situations on signature validation (instead of validation report, error message is returned)

Specific test cases and input files can be found in:

• StatisticsToLogsManualIT.java

SiVa Demo Application tests

Testing of SiVa Demo Application is done manually. The main cases are:

• Cross browser usage (IE, Edge, Chrome, Firefox and Safari)
• File upload (different sizes, suported and unsupported file types)
• Displayment of Validation Report both for REST and SOAP
• Layout of the page
• Error representation

Sample test cases with input files can be found in:

• Demo Application Integration Test

System Test introduction

While Integration Tests were mostly carried out automatically then System Testing is mostly depending on manual testing.

System testing is carried out using two access points:

• Testing through SiVa Demo Application
• Testing through X-Road security server using SoapUI

Note

Testing through X-Road security server requires presence and configuration of X-Road security server to use SiVa service. Tests are run using SoapUI that simulates request to X-Road security server.

Testing through X-Road security server

Following areas are covered for document validation:

• Validation of valid signature
• Validation of invalid signatur
• Validation that returns Soap error

All of the above test cases are run with BDOC, DDOC, PDF and ASIC-S containers.

Following areas are covered for file extraction:

• Extraction of data files from valid ddoc
• Extraction of data files from invalid ddoc
• Error response from data file extraction

Tests along with test case descriptions are available for rerun in github.

Specific test cases and input files can be found in:

• X-Road Soap System Test

Configuration/administration testing

Following areas are covered:

• SiVa Web Application configuration
• Demo application configuration

Specific test cases can be found in:

• Configuration System Test

Performance testing

More information can be found in SiVa-perftests GitHub page.