# Testing Framework
This is a set of scripts that roughly does the following:
- launches available libvirt VMs on host
- synchronizes code and test code to each VM
- runs a build script on each VM
- runs a test script on each VM
- collects job status and logs and stores them locally

## Prerequisites
- SECRET.sh must contain the following
```sh
USERN= # username on your VMs
PASSP= # password for said user
VM_PARALLEL= # number of CPU cores a VM has
HOST_PARALLEL= # number of CPU cores to build with locally on the hypervisor
```
- sshpass, virsh, libvirt, etc
- cloned repos of nginx, nginx-tests, and nginx-otel

## About those VMs....
- hostname and libvirt domain name need to be same for each
- username and password should be the same on all of them
- whatever your test functions need (see `nginx.sh`)
  - git
  - make
  - gcc
  - zlib
  - pcre
  - openssl
  - rsyncz
  - perl and perl-utils (for prove)

For Otel module build and tests:
  - cmake
  - c-ares
  - linux-headers
  - g++ / clang++ / etc
  - python3, pytest, pip
  - protobuf, grpc

### FreeBSD
- need to install bash
- need to set login shell to bash
- need to install pkgconfig

### Fedora
- install zlib-ng-compat-devel and zlib-ng-compat-static for zlib, not zlibrary-devel or zlib-ng-devel.
- fedora also seems to need the openssl-devel-engine package.
- need perl-Test-Harness and perl-FindBin and perl-Sys-Hostname for the nginx tests

## Usage
Invoke `test.sh` with some or all of the following flags:
- `--nginx <nginx>` takes an nginx code directory and builds it on remote hosts
- `--otel <otel>` takes an nginx-otel directory and builds it on remote hosts
- `--tests <tests>` takes an nginx-tests directory and runs tests on remote hosts
    requires that `--nginx=...` also be supplied. If otel was supplied this also
    triggers testing in otel directory.

Logs are in logging directory shown. They are split out into files per VM per phase.
User may set test_log_dir to provide their own logging directory.

When running, you may view the combined status using a command like one of the following.
The logging dir will be specified in the first second of output from the test.sh script.
```sh
tail -f <LOGGING DIR>/on*
tail -f <LOGGING DIR>/sync*
tail -f <LOGGING DIR>/build_nginx*
tail -f <LOGGING DIR>/build_otel*
tail -f <LOGGING DIR>/test_nginx*
tail -f <LOGGING DIR>/test_otel*
tail -f <LOGGING DIR>/off*
```

Additionally, virt.sh and nginx.sh can be sourced in a shell profile so that their
functions can be used at terminal prompt. For example, sourcing virt.sh will provide
the `vsh` function which turns on a VM, opens an SSH connection to it, and then turns it off again. Functions such as `vms_avail`, `vms_on`, and `vms_off` may also be of use.

In the future, routines will be added to nginx.sh that automate several local development and testing processes.

# Contributions
Any contributions which make the procedures defined in these scripts portable across
shells and platforms is welcomed. Any additional feature add for any enclosed script
is also welcomed.