Writing Unit Tests for Galaxy

In this post I will be examining and reflecting the processes which must be undergone in order to write a test case for Galaxy, from beginning to end.

The basic rundown of the calling hierarchy:

1. sh runAllTests.sh
2. testCaseX.py (testCasesExecutables/)
3. testCaseX.txt (testCases/)  
4. testCaseDatasets/
5. testCaseXReport.txt (temp/) and testCaseXOracle.txt (oracles/)
6. Team3_Testing_Report.html (reports/)

1. Running all the test cases
There is a script located in the scripts/ folder known as runAllTests.sh. This script first removes the last testing report that has been produced. In the future, this will be modified to also delete test case reports, but more on that later. Second, the script iterates through every file in the testCasesExecutables/ folder and runs them using a for-each loop with a python call. It is worth noting that this also runs our helper file, but, because it is only a declaration of functions, it does not interfere with any calls or act as a detriment to timing for these tests. So within each call of a test case executable file we have to consider how we will be...

2. Running the executable
The first executable that will be called is testCase1.py. This file looks very clean as we have abstracted a lot of work that would be repeated in every test case file. We make the imports to subprocess, os, and testCaseFunctions. Subprocess is a module that we can import to make command line calls, a pretty vital element to testing from the command line. The invocation using subprocess is seen toward the end of the executable where we use the line:

# Let's make the call to the shell
subprocess.call(testCaseScript,shell=True)

This allows us to make a call directly to the shell - this is a security issue, but this is a project of ours we are running on open-source software, so this is not too much of a worry for us.

We make a call to os in order to have a default directory we are making calls from - this allows us to write relative paths in a more readable manner. This is done with the following line: 

# Change the working directory to Galaxy's Home section to make our calls simpler (this also enforces our helper method calls)
os.chdir('../project/src/galaxy-dist/')

Lastly, we call testCaseFunctions in most other lines of the file. This is to interpret the textual version of the test case, to assemble our command line call, validate results with our testing oracles, and, finally, produce a professional-grade testing report.

3. Textual test case

Located within the testCases/ folder, our textual test cases are a vital part of guiding our test cases. For explanation, I will be referring to testCase1.txt. There exists full documentation about this file within the README within the Docs/ folder of our project, but I will detail a very basic version of the rundown. 

Test ID:

A test ID is a textual description of the test being managed at that point. This ends up being passed to the final html. This ID is unique because no test case should be the same as a test case that already exists (but there are not preventative measures stopping someone from adding a congruous test case).

Requirement Being Tested:

This section names the function we are dealing with (for HTML purposes) along with the atomic requirement being handled. In our example, this specific requirement is: "Calculate the length of a single fasta-encoded string. Length = 1."

Component Being Tested:

Galaxy's hierarchy very easily allows us to pick out what component is being tested with this specific requirement. Galaxy has toolsets which are made up of various tools. This tools are, typically, one function tools (in this example, "fasta_compute_length" is one tool that performs one very basic function within the "fasta_tools" toolset). This allows us to pick out the name being managed and even use it when building relative directories into our test cases.

Method Being Tested:

This details the actual method that we are handling. In our example, the entire tool is one method and it is the one we are testing. 

Test Inputs:

Test inputs make up the instantiation of the method we are testing. There is a line for each set of inputs we would need in order to test a tool and this is all handled with Python scripting. The order of the inputs is very important, though, as we have to build command-line arguments with these inputs and misordering them can confuse the command line and produce errors or false results.

Expected Outcome:

Expected outcome details which oracle we should refer to whenever comparing our testing report. Additionally, there is a textual description of what we expect to see under the oracle file.

4. Test case datasets

The nature of Galaxy lends itself to using full-on datasets rather than small inputs. Galaxy is meant to be use for data-intensive biology, but there is nothing stopping us from using the methods and input types of large biology datasets with our own test cases. In fact, this is the only way to effectively conduct tests of this nature. For example, within our testCaseDatasets/ folder, we have, currently, 5 FASTA files that contain data that is interpreted by the individual test cases. Following with the example we were using earlier, the testCase1.fasta file is a very simple file that contains a FASTA ID (no restrictions other than having to start with a ">") and then an encoded string on the very next line. The test case for the first test is about measuring a FASTA string that has only one character, hence only the value "C" being in the string section of that FASTA file.

5. Textual test case reports

Each individual test that is executed creates a textual report within the temp/ directory. Currently, each time a test is run, the new test case report will overwrite the old test case report. This is effective and it works, but we are going to work on getting the runAllTests.sh file to also remove all files within this directory (which should be easy with a rm -R call). These reports are nothing more than the results of running the tool with our given inputs. These files are what are compared to the oracle files. The files have to be exactly the same or else there will be a result that ends with a failed test.

6. Full testing report

While we compare each of those textual test cases to their oracles, we are slowly building up an HTML report for a final result within our reports/ section. This is done by using a htmlBackbone.html file we have obtained and appending individual testing results to our full report. The backbone came from running one of Galaxy's built-in functional test and then we stripped out all of the nonsense that was not CSS. This way, we can use Galaxy's CSS while using our tests and create a professional-grade report that mirrors the look and feel of Galaxy.

Music listened to while blogging: Spotify radio station based off of C90s - Shine a Light (Flight Facilities Remix)