This vignette steps through the process of validating external resources with {valtools}. “External resources” is defined for this vignette as any software or environment that does not follow the validation process defined in the Starting New Validation Package using {valtools} vignette. This means packages installed from other sources, collections of packages, external software, and environments.
This process may be invoked when an R package was already developed and only after the fact was validation decided to be of value, validating an externally generated package (for example a package installed from CRAN), or validating an environment or system.
Starting a new validation packet using {valtools} starts with
vt_create_packet(). Here, the user passes the directory the
project is to be performed in, and optionally the target of
validation.
The “target of validation” is what the packet intends to validate - be it an external package, environment, software, etc.
If the target is not provided, {valtools} will request the user provide this information
# create a new validation packet skeleton
vt_create_packet(
path = "/my/path/to/example_validation_packet",
target = "super.package"
)The validation packet folder is constructed using the
usethis::create_project() function, and then all the
necessary validation infrastructure required for {valtools}.
Importantly, there is now a validation folder, which is
the working directory for the validation. This is where almost all the
content for validation will be created.
Inside this folder, there is the validation.yml file,
which will be referred as the validation config file going forward. This
YML file informs {valtools} how to interact with the various pieces of
validation that will be created, and information that needs to be shared
across multiple sessions/users. The user does not need to interact with
this file directly, the functions inside {valtools} will update this
file as necessary.
Run the chunk of code below to create a validation packet in a temporary directory to follow along with the tutorial for the {whoami} package. The new packet project will be opened in a new session. Run all subsequent code in that new session.
valtools::vt_create_packet(
path = file.path(tempdir(),"validation_packet_whoami"),
target = "whoami"
)To examine the folder structure of the new package, run the following function:
Requirements document users needs of the target - what are the
problems the target solves for the users - and must be documented before
any test cases are written. Requirements are recorded within the
validation/requirements folder by default. The collection
of requirements may be called specifications.
Requirements are written as markdown (.md) documents with special
roxygen headers. Each requirement must have the following roxygen
comments in the header: title, editor, editDate, and riskAssessment. The
last the roxygen comments are custom {valtools} supported roxygen tags
to support validation. @editor is for tracking the last
editor of the function, @editDate is for recording whenever
a function is modified, and @riskAssessment is for tracking
risks for each requirement.
To make adding validation content easy, {valtools} extended the
usethis approach to package contents creation through a
family of “vt_use_*” functions.
vt_use_req() creates a new requirement in the
validation/requirements folder, with the main argument
being the name of the requirement, and an optional argument
username to record the name of the person writing the
requirement.
If the username argument is not passed, {valtools} will
automatically get the computer username of the user creating the
requirement and attempt to put in their full name. If the user has not
created any validation contents before, it will ask the user some
questions (Name, Title, and Role) and record them in the validation
config file for documentation in the validation report.
Run the command above and in the newly opened requirements file, on
line 5, Replace REQUIREMENTS with 1.1, and
ASSESSMENT with 1, Low Risk, Small Impact to
indicate requirement 1.1 has a risk assessment that determined it has a
low risk and small impact when it is wrong.
Add a new line underneath the line above (at line 6) line that
contains: #' 1.2: 5, Low risk, Medium Impact
Copy the following content:
+ 1.1 Collect user id for current session
+ 1.2 Collect full name of user for current sessionSimilar to a news file, {valtools} suggests the use of a change log that is directly tied to validation for recording changes. The purpose of this is to capture update and information that is useful for developers from information that is important to capture in validation.
To create this change log file, {valtools} has the function
vt_use_change_log(). It will create the change log file
inside the working directory and open it up for editing.
The header information tracks the version of validation and the date of the release of validation. This is a markdown file, so normal markdown can be used to document the changes. However, critically here, only bullets marked with [validation] will be recorded in the validation report.
Run the command above to create a change log.
Testing is done to ensure that the target meets the requirements that were set out for the project. Testing is done in two major steps: the firsts consists of writing out a series of cases that would prove that the requirements have been met, the second is the application of these cases.
The addition and writing of test cases is handled by the
vt_use_test_case() function. Similarly to
vt_use_req(), a username can be passed, or it will look to
determine which user is calling the function and input their
information.
This function creates the test case file in the
validation/test_cases folder of the package and opens it
for editing.
Run the code above and in the newly opened test case file, replace
TESTCASE with 1.1, and
REQUIREMENT with 1.1 to indicate test case 1.1
shows that requirement 1.1 is being met.
Add a new line underneath the line above (at line 6) line that
contains: #' 1.2: 1.1, 1.2
This is to indicate test case 1.2 shows requirements 1.1 and 1.2 are being met.
Copy the following test case into file where test cases are to be documented:
+ 1.1 Test that the software can identify the username pf the user by
setting the environment variable `LOGNAME` to `jsmith` for the duration of
the test and confirming that the output of whoami::username is `jsmith`.
+ 1.2 Test that the software can identify the full name of the user by
setting the environment variable `FULLNAME` to "John Smith" for the
duration of the test and confirming that the output of whoami::fullname()
is "John Smith"Test code is the implementation of the test cases as code. The goal is that the code is completely reproducible and able to be run without human interaction. Additionally, test code is written by a third party - someone that was not involved with writing the actual code or the test case. This helps ensure the integrity of the testing as well as providing valuable review of the documentation of the test cases and package code.
Similarly to vt_use_req() for requirements and
vt_use_test_case for test cases, {valtools} provide a
function for creating test code files and recording which user created
the file.
Add “Val A Dashun” to the validation config file:
valtools::vt_add_user_to_config(
username = "user_b",
name = "Val A Dashun",
title = "Programmer II",
role = "tester"
)Now that this persons information is recorded, construct the test code file that they will use to record the test code through the code below.
In the newly opened test code file. Update TESTNUMBER to
1.1 in the new test code file and copy the code below into
the body of the test:
withr::with_envvar(
new = list(LOGNAME = "jsmith"),
{
user <- whoami::username()
expect_equal(
user,
"jsmith"
)
})add a new test with the following beneath the test. Replace “TODAYS DATE” with today’s date.
Now that there is a validation report as an Rmarkdown, validation is only a compiling of the report away. To validate the target, we execute the report non-interactively and save the results.