---
title: "Validating External Resources with {valtools}"
output: 
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 3
vignette: >
  %\VignetteIndexEntry{Validating External Resources with {valtools}}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  markdown: 
    wrap: 72
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval=FALSE
)
```

```{r setup, warning=FALSE}
# remotes::install_github("phuse-org/valtools")
library(valtools)
```

This vignette steps through the process of validating external resources with
{valtools}. "External resources" is defined for this vignette as any software 
or environment that does not follow the validation process defined in the 
[Starting New Validation Package using {valtools}](starting-validated-package.html) 
vignette. This means packages installed from other sources, collections of packages,
external software, and environments.

This process may be invoked when an R package was already developed and 
only after the fact was validation decided to be of value, validating an 
externally generated package (for example a package installed from CRAN), or 
validating an environment or system.


Creating A New Validation Packet
------------------------

Starting a new validation packet using {valtools} starts with `vt_create_packet()`.
Here, the user passes the directory the project is to be performed in, and optionally 
the target of validation. 

The "target of validation" is what the packet intends to validate - be it an 
external package, environment, software, etc. 

If the target is not provided, {valtools} will request the user provide this 
information


```{r}

# create a new validation packet skeleton
vt_create_packet(
  path = "/my/path/to/example_validation_packet", 
  target = "super.package"
  )

```

The validation packet folder is constructed using the `usethis::create_project()`
function, and then all the necessary validation infrastructure required for
{valtools}.

Importantly, there is now a `validation` folder, which is the
working directory for the validation. This is where almost all the
content for validation will be created.

Inside this folder, there is the `validation.yml` file, which will be
referred as the validation config file going forward. This YML file
informs {valtools} how to interact with the various pieces of validation
that will be created, and information that needs to be shared across
multiple sessions/users. The user does not need to interact with this
file directly, the functions inside {valtools} will update this file as
necessary.

#### Tutorial

Run the chunk of code below to create a validation packet in a temporary 
directory to follow along with the tutorial for the {whoami} package. The new 
packet project will be opened in a new session. Run all subsequent code in 
that new session.

```{r}

valtools::vt_create_packet(
  path = file.path(tempdir(),"validation_packet_whoami"),
  target = "whoami"
  )

```

To examine the folder structure of the new package, run the following
function:

```{r}

fs::dir_tree(recurse = TRUE)

```

Add Requirements
----------------

Requirements document users needs of the target - what are the problems the target
solves for the users - and must be documented before any test cases are written.
Requirements are recorded within the `validation/requirements` folder by default.
The collection of requirements may be called specifications.

Requirements are written as markdown (.md) documents with special roxygen headers. 
Each requirement must have the following roxygen comments in the header: title,
editor, editDate, and riskAssessment. The last the roxygen comments are custom
{valtools} supported roxygen tags to support validation. `@editor` is for tracking
the last editor of the function, `@editDate` is for recording whenever a function is
modified, and `@riskAssessment` is for tracking risks for each requirement.

To make adding validation content easy, {valtools} extended the
`usethis` approach to package contents creation through a family of
"vt\_use\_\*" functions.

`vt_use_req()` creates a new requirement in the
`validation/requirements` folder, with the main argument being
the name of the requirement, and an optional argument `username` to
record the name of the person writing the requirement.

If the `username` argument is not passed, {valtools} will automatically
get the computer username of the user creating the requirement and
attempt to put in their full name. If the user has not created any
validation contents before, it will ask the user some questions (Name,
Title, and Role) and record them in the validation config file for
documentation in the validation report.

```{r}

valtools::vt_use_req("Requirement_001")

```

#### Tutorial

Run the command above and in the newly opened requirements file, on line
5, Replace `REQUIREMENTS` with `1.1`, and `ASSESSMENT` with
`1, Low Risk, Small Impact` to indicate requirement 1.1 has a risk
assessment that determined it has a low risk and small impact when it is
wrong.

Add a new line underneath the line above (at line 6) line that contains:
`#' 1.2: 5, Low risk, Medium Impact`

Copy the following content:

    + 1.1 Collect user id for current session
    + 1.2 Collect full name of user for current session

Change Log
----------

Similar to a news file, {valtools} suggests the use of a change log that
is directly tied to validation for recording changes. The purpose of this is to
capture update and information that is useful for developers from
information that is important to capture in validation.

To create this change log file, {valtools} has the function
`vt_use_change_log()`. It will create the change log file inside the
working directory and open it up for editing.

The header information tracks the version of validation and the date of the
release of validation. This is a markdown file, so normal markdown can
be used to document the changes. However, critically here, only bullets
marked with [validation] will be recorded in the validation report.

```{r}

valtools::vt_use_change_log()

```

#### Tutorial

Run the command above to create a change log.

Testing
-------

Testing is done to ensure that the target meets the requirements that
were set out for the project. Testing is done in two major steps: the
firsts consists of writing out a series of cases that would prove that
the requirements have been met, the second is the application of these
cases.

### Test Cases

The addition and writing of test cases is handled by the
`vt_use_test_case()` function. Similarly to `vt_use_req()`, a username
can be passed, or it will look to determine which user is calling the
function and input their information.

This function creates the test case file in the
`validation/test_cases` folder of the package and opens it for
editing.

```{r}

valtools::vt_use_test_case("Test_case_001")

```

#### Tutorial

Run the code above and in the newly opened test case file, replace
`TESTCASE` with `1.1`, and `REQUIREMENT` with `1.1` to indicate test
case 1.1 shows that requirement 1.1 is being met.

Add a new line underneath the line above (at line 6) line that contains:
`#' 1.2: 1.1, 1.2`

This is to indicate test case 1.2 shows requirements 1.1 and 1.2 are
being met.

Copy the following test case into file where test cases are to be
documented:

    + 1.1 Test that the software can identify the username pf the user by
    setting the environment variable `LOGNAME` to `jsmith` for the duration of
    the test and  confirming that the output of whoami::username is `jsmith`.
    + 1.2 Test that the software can identify the full name of the user by 
    setting the environment variable `FULLNAME` to "John Smith" for the
    duration of the test and confirming  that the output of whoami::fullname()
    is "John Smith"

### Test Code

Test code is the implementation of the test cases as code. The goal is
that the code is completely reproducible and able to be run without
human interaction. Additionally, test code is written by a third party -
someone that was not involved with writing the actual code or the test
case. This helps ensure the integrity of the testing as well as
providing valuable review of the documentation of the test cases and
package code.

Similarly to `vt_use_req()` for requirements and `vt_use_test_case` for
test cases, {valtools} provide a function for creating test code files
and recording which user created the file.

```{r}

valtools::vt_use_test_code("Test_code_001")

```

#### Tutorial

Add "Val A Dashun" to the validation config file:

```{r}

valtools::vt_add_user_to_config(
  username = "user_b",
  name = "Val A Dashun",
  title = "Programmer II",
  role = "tester"
)

```

Now that this persons information is recorded, construct the test code
file that they will use to record the test code through the code below.

```{r}

valtools::vt_use_test_code("Test_code_001", username = "Val A Dashun")

```

In the newly opened test code file. Update `TESTNUMBER` to `1.1` in the
new test code file and copy the code below into the body of the test:

``` {.r}

withr::with_envvar( 
  new = list(LOGNAME = "jsmith"),
  {
  
  user <- whoami::username()
  
  expect_equal(
  user,
  "jsmith"
 )
 
})

```

add a new test with the following beneath the test. Replace "TODAYS
DATE" with today's date.

``` {.r}

#' @editor Val A Dashun
#' @editDate TODAYS DATE
test_that("1.2",{

  withr::with_envvar( 
    new = list(FULLNAME = "John Smith"),
    {
    
    user_full_name <- whoami::fullname()
    
    expect_equal(
    user_full_name,
    "John Smith"
   )
  
  })
})
```

Authoring Validation Reports
--------------------------------

{valtools} provides dynamic access via a Rmarkdown file to details necessary for 
generating a validation report at push of button. This validation report documents
that the package meets stated goals and can be re-evaluated as necessary to 
generate the report in PDF or HTML format. 

The function `vt_use_report()` creates a validation report rmarkdown file 
pre-populated with code to scrape all the pieces of information
that were generated in the prior steps to create the final report when being knit.

`vt_use_report()` saves the validation report rmarkdown file 
in the working directory identified in the validation config file. 
Within packages this defaults to the base folder.
This rmarkdown file will have a default name `validation.Rmd` if unspecified.

```{r}
valtools::vt_use_report(template = "packet")
```

There are several sections included by default in the provided validation report 
rmarkdown:

-   Signatures: Capture signatures of everyone involved in the
    validation.

-   Release Details:

    -   Records the validation environment
    -   Presents the change log of the validation.
    -   Subsections to show the last editor for each piece of the
        validation; requirements, test cases and test code.
    -   Traceability table to show which requirements are being
    -   Risk Assessment: Combines all the risk assessments made into a
        single table

-   Validation: record each requirement, test case, and results of the
    test code

{valtools} also supports a concept called "dynamic referencing", which
will be explained in another vignette.

When editing the report, some key functions to know for extending the
report included by {valtools} are:

-   `vt_path()` allows user to base path from the validation directory.
    Similar idea to the {here} package, but for validation.
-   `vt_file()` allows the user to point to specific files and render
    them as child documents within the report.
-   `vt_scrape_*` family of functions allows users to scrape various
    pieces of information from the validation infrastructure and returns
    a data.frame.
-   `vt_kable_*` family functions provides an opinionated formatting to
    the `vt_scrape_*` functions to help quickly construct the report.
-   `vt_get_child_files()` returns the list of files that are indicated
    in the validation.yml to be included in the validation report. This
    allows for batch creation of the dynamic content in the report.

Keep in mind, the report is an Rmarkdown, so there is no limit to
editing and customization, and templates.

#### Tutorial

Run the code above to generate the report, and inspect the overall
structure of the report. See what happens when contents are moved
around.

Running a Validation Report
---------------------------

Now that there is a validation report as an Rmarkdown, validation is only a
compiling of the report away. To validate the target, we execute the report 
non-interactively and save the results.

#### Tutorial

Run the validation report

##### Validation Mode: Running on Source

```{r}
valtools::vt_validate_report()
```