Deliverable #3 Experience

I'd like to start this post by showcasing the logo I made for my software engineering team, the Boolean Bombers, using Logo Garden. The logo can be seen on the right.

There were a few criticisms of our current testing suite for Galaxy. These were very quick, very easy fixes. I took the liberty to make the corrections.

For starters, we were creating a Python file for each test case, which is not what we should have been doing. What is funny about this, though, is that when we were creating these python files, the only difference was the filename that we were using to find the testCaseX.txt file. So it was very easy to create a loop that runs all the code for each testCaseX.txt file within the testCases/ directory. This avoids storing loops and works regardless of how many or how few test cases we have.

Additionally, I added a quick boolean to check if we even had a test case to begin with. If we didn't have a test case, then we jump into an if statement at the end of the driver that reports to the html that there were no tests.

On top of all this, I added a quick line of code to the runAllTests.sh script that opens the browser with test results and, if a browser is already open, then it opens the results in a new tab in that browser.

To add to that, I made some quick changes in the CSS in the HTMLBackbone that retains styling to keep the namesake for testing (pass vs fail, rather than success vs fail/error).

All things considered, we are in excellent shape for the rest of this project.

Aside: I am a part of a Spotify internship right now and one of the things we have to do is try and refer people to Spotify. There's a contest going on right now where we can earn rewards based on who gets the most people referred. My referral URL is here. 

Music listened to while blogging: Donald Glover comedy album ~ Weirdo

Galaxy Deliverable #3

For this post, I will be reflecting upon the final edits of my team's Galaxy Testing Project before our next deadline (tomorrow).

For starters, the testing framework did not remove files from the temp directory. The temp directory held all the reports that are produced by tests (1 report per test) that are then compared to the oracles. The framework now does this through the following call at the beginning of runAllTests.sh:

rm ../temp/*.*

The reason we do it this way rather than doing a recursive remove of the temp directory is so that we preserve the existence of the temp directory. We very easily could have taken an alternate route where we do the following:

rm -r ../temp

mkdir ../temp

These two segments of code achieve the same goal in the end.

Lastly, the testCaseX.txt files were cleaned up. Before, the "Expected Outcome:" section contained the name of the oracle file and the contents of the oracle file. This was a bit too redundant for my taste, so I removed the contents of the expected outcome from the testCaseX.txt file. The Python code did not have to be edited here since we checked the oracle for the actual results anyways. 

What we did change in this part of the framework, though, was the way in which oracles were found. Before, the validation of test results through the oracle was performed through the invocation of a method I created that took a test case number and then created a call to an oracle. Instead, the testCaseX.txt file is read and the actual oracle filename is read and then sought out by the validation method.

Music listened to while blogging: N/A

Learn2Mine and Skill Trees

This post will not be reflective of my team's ongoing Galaxy Testing project as we have not met since my last post (we will be meeting and working tomorrow morning). So, instead, I will detail my latest work on my own research in the Anderson Lab here at the College of Charleston.

Learn2Mine utilizes gamification in order to help teach students data science. My latest work on the application is on revamping the way in which users view their skill tree.

Fig. 1 - Old Skill Tree

The best way to talk about these concepts is to use the physical images. The initial skill tree implementation is able to be seen in figure 1. This skill tree used a regular block method to represent skills that users have unlocked, learned, or mastered. This method seems a bit too cut and dry, though. For starters, we had 2 skill trees with this current implementation to represent pattern recognition techniques in one tree and R programming skills in a different, second tree. Also, if a skill was 'locked', then users were not able to work with that skill.

Fig. 3 - Classification Subtree

Fig. 2 - New Skill Tree Prototype

Now we have departed from that. Figure 2 shows the entirety of our current skill tree prototype - this prototype is created through a javascript implementation of Cytoscape (a flash-based graphical environment). We give a  hierarchical progression of skills that are represented in tree-form. This is not strict, though; this hierarchy serves more as a guideline for what order we feel users will learn the best. There are many different progressions one can take through the skill tree. For example, you can go down the classification branch in the tree (as seen in figure 3). This will have you learning the basics of the K-Nearest Neighbor algorithm, Partial Least Squares Regression, and Neural Networks. We also have a mastery lesson for K-Nearest Neighbors that requires users to implement optimization techniques in order to improve their KNN classification over a specific threshold.

We make new lessons just about every week for Learn2Mine, as it is going hand-in-hand with Dr Anderson's Data Science 101 class. This is functioning as a replacement for other workflow management systems that implement data science techniques. Typical systems used in DATA 101 include Weka and RapidMiner - these systems have lots of flaws, flaws that we are trying to combat with our system, while also focusing on the educational aspect of this process. This launch in the DATA 101 class is also functioning as our pilot test deployment of the system.

Music listened to while blogging: 50 Cent and J. Cole

Further Information on Unit Testing with Galaxy

There has not been much headway with the Galaxy testing project since my last blog post, but we did run into some hurdles we did not foresee.

Our original implementation of our unit testing architecture banked on the usage of datasets. This seemed natural since Galaxy always uses datasets when doing its testing. The reason for this is because Galaxy's primary use is for dealing with large biological datasets, which naturally lend themselves to dataset files, rather than manual input. Since we are testing such small pieces of the architecture and tools, it may be more helpful for us to input actual values rather than forcing a dataset constraint.

In Python, this is extremely easy. Within our testCaseX.txt file, we just need to specify some kind of delimiter or boolean that can be programmatically read. The presence (or absence) of this value could mean just read the raw data value out of the testCaseX.txt file, rather than finding a dataset in a different folder. This will allow for ease when adding future tests. This also adds an extra way to break tests and see if they fail and break down as expected.

This, and much more, should be implemented before my next update here and I will reflect on those changes, along with the associated obstacles, in my next post. Consider all things, this project seems to be moving along smoothly, even though the team dynamic feels a bit lacking.

Music listened to while blogging: Kendrick Lamar and The C90s

Writing Unit Tests for Galaxy

In this post I will be examining and reflecting the processes which must be undergone in order to write a test case for Galaxy, from beginning to end.

The basic rundown of the calling hierarchy:

1. sh runAllTests.sh
2. testCaseX.py (testCasesExecutables/)
3. testCaseX.txt (testCases/)  
4. testCaseDatasets/
5. testCaseXReport.txt (temp/) and testCaseXOracle.txt (oracles/)
6. Team3_Testing_Report.html (reports/)

1. Running all the test cases
There is a script located in the scripts/ folder known as runAllTests.sh. This script first removes the last testing report that has been produced. In the future, this will be modified to also delete test case reports, but more on that later. Second, the script iterates through every file in the testCasesExecutables/ folder and runs them using a for-each loop with a python call. It is worth noting that this also runs our helper file, but, because it is only a declaration of functions, it does not interfere with any calls or act as a detriment to timing for these tests. So within each call of a test case executable file we have to consider how we will be...

2. Running the executable
The first executable that will be called is testCase1.py. This file looks very clean as we have abstracted a lot of work that would be repeated in every test case file. We make the imports to subprocess, os, and testCaseFunctions. Subprocess is a module that we can import to make command line calls, a pretty vital element to testing from the command line. The invocation using subprocess is seen toward the end of the executable where we use the line:

# Let's make the call to the shell
subprocess.call(testCaseScript,shell=True)

This allows us to make a call directly to the shell - this is a security issue, but this is a project of ours we are running on open-source software, so this is not too much of a worry for us.

We make a call to os in order to have a default directory we are making calls from - this allows us to write relative paths in a more readable manner. This is done with the following line: 

# Change the working directory to Galaxy's Home section to make our calls simpler (this also enforces our helper method calls)
os.chdir('../project/src/galaxy-dist/')

Lastly, we call testCaseFunctions in most other lines of the file. This is to interpret the textual version of the test case, to assemble our command line call, validate results with our testing oracles, and, finally, produce a professional-grade testing report.

3. Textual test case

Located within the testCases/ folder, our textual test cases are a vital part of guiding our test cases. For explanation, I will be referring to testCase1.txt. There exists full documentation about this file within the README within the Docs/ folder of our project, but I will detail a very basic version of the rundown. 

Test ID:

A test ID is a textual description of the test being managed at that point. This ends up being passed to the final html. This ID is unique because no test case should be the same as a test case that already exists (but there are not preventative measures stopping someone from adding a congruous test case).

Requirement Being Tested:

This section names the function we are dealing with (for HTML purposes) along with the atomic requirement being handled. In our example, this specific requirement is: "Calculate the length of a single fasta-encoded string. Length = 1."

Component Being Tested:

Galaxy's hierarchy very easily allows us to pick out what component is being tested with this specific requirement. Galaxy has toolsets which are made up of various tools. This tools are, typically, one function tools (in this example, "fasta_compute_length" is one tool that performs one very basic function within the "fasta_tools" toolset). This allows us to pick out the name being managed and even use it when building relative directories into our test cases.

Method Being Tested:

This details the actual method that we are handling. In our example, the entire tool is one method and it is the one we are testing. 

Test Inputs:

Test inputs make up the instantiation of the method we are testing. There is a line for each set of inputs we would need in order to test a tool and this is all handled with Python scripting. The order of the inputs is very important, though, as we have to build command-line arguments with these inputs and misordering them can confuse the command line and produce errors or false results.

Expected Outcome:

Expected outcome details which oracle we should refer to whenever comparing our testing report. Additionally, there is a textual description of what we expect to see under the oracle file.

4. Test case datasets

The nature of Galaxy lends itself to using full-on datasets rather than small inputs. Galaxy is meant to be use for data-intensive biology, but there is nothing stopping us from using the methods and input types of large biology datasets with our own test cases. In fact, this is the only way to effectively conduct tests of this nature. For example, within our testCaseDatasets/ folder, we have, currently, 5 FASTA files that contain data that is interpreted by the individual test cases. Following with the example we were using earlier, the testCase1.fasta file is a very simple file that contains a FASTA ID (no restrictions other than having to start with a ">") and then an encoded string on the very next line. The test case for the first test is about measuring a FASTA string that has only one character, hence only the value "C" being in the string section of that FASTA file.

5. Textual test case reports

Each individual test that is executed creates a textual report within the temp/ directory. Currently, each time a test is run, the new test case report will overwrite the old test case report. This is effective and it works, but we are going to work on getting the runAllTests.sh file to also remove all files within this directory (which should be easy with a rm -R call). These reports are nothing more than the results of running the tool with our given inputs. These files are what are compared to the oracle files. The files have to be exactly the same or else there will be a result that ends with a failed test.

6. Full testing report

While we compare each of those textual test cases to their oracles, we are slowly building up an HTML report for a final result within our reports/ section. This is done by using a htmlBackbone.html file we have obtained and appending individual testing results to our full report. The backbone came from running one of Galaxy's built-in functional test and then we stripped out all of the nonsense that was not CSS. This way, we can use Galaxy's CSS while using our tests and create a professional-grade report that mirrors the look and feel of Galaxy.

Music listened to while blogging: Spotify radio station based off of C90s - Shine a Light (Flight Facilities Remix)

Deliverable 2 Meta-Experience

From my title you should be able to deduce that the bulk of this blog post will be about the experience I, and my team, had whenever we presented our second deliverable for our testing project.

What I thought was going to be a glorious experience turned out to be one of the more horrific experiences I have had. I thought I had set my team up to be pretty far ahead in our project by already coding up our test cases and having HTML output. Apparently, though, the way I was conducting the tests did not follow the exact specifications for my Software Engineering class. I incorporated a couple of the file calls from Galaxy into my test cases and apparently that is not allowed. We have to, instead, reinvent the wheel. I was improving upon Galaxy's testing suite by implementing a way to conduct individual tests from individual tools (something they have listed in their documentation and wiki as something they need to do). Instead, though, we have to develop a testing suite that stands alone from the Galaxy project. The way I was doing it would be great to have submitted to Galaxy, but it would require too much work to do that and fit the specifications for the Software Engineering team project. While this is disappointing, this is the work that I have been given and, much like in the real-world when working for a company, the work given by the customer is the work that needs to be done.

All in all, this has been a good learning experience, though frustratingly wasteful for me when considering the amount of time I dumped into the functional testing.

The next step for my group and me is to examine the individual units that underlie the functions we have been examining. We have been examining the "Upload Data" toolset and this toolset has a multitude of helper functions that it requires in order to run. By extracting these helper functions, we can test each of them individually which, in essence, is a parallelization of unit testing as this goes hand-in-hand with it. If we can establish unit tests that work on these helper functions, then we may be able to work our way all the way up to testing a function that utilizes these test cases, which, in essence, is creating a functional test. In my eyes, this would be an immense, intensive learning experience that everyone in my team can benefit from.

Music listened to while blogging: The C90s

Deliverable 2 Experience

For starters, this second deliverable tripped my team and me up quite a bit.

Initially, I thought we had to implement 5 test cases (not just identify them) so I took the leap and created those test cases and the HTML output to follow. This is a requirement for the third deliverable of this testing project so at least I am ahead of the game on that.

Subversion, however, has decided that we have to become mortal enemies. When I initially downloaded our team repository, I ended up pulling each of my team's individual development branches in addition to the trunk, which we tried to avoid. We did this so we can simply do merges when we were confident with our changes. We wanted to test our changes and tests before merging our branches with the head of the master branch.

When I tried to push my final testing scripts into Subversion my GUI, RapidSVN, keeps mentioning "Unknown Error!" and a 405 error, which they really do not detail much. Upon short searching, I found that some people were able to fix this problem by re-checking out the repository and then proceeding to make changes. I am currently doing this, but Subversion likes to take its time and take hours to pull down development branches so I am currently waiting on that to finish.
Update: I just made it so I only pulled out the folder I need from the repository and Subversion finished the checkout in a matter of minutes. I then committed my changes up and they can now be viewed in the team repository (link on the right side of the page).

My goal is to get these scripts up before our presentation of our second deliverable tomorrow. I created python files that make a BASH call to a built-in Galaxy method that allows the running of functional tests. I decided to make the call this way because Galaxy creates a minimal HTML file with some important data about the test. So we developed tests and pass them to this BASH call.

The overarching structure:

runAllTests.sh loops and makes calls to each of the 5 test cases
The first test case generates the initial html file after the test finishes running
The proceeding test cases append to that html when their respective tests finish (Using Python to generate HTML code)
Once the runAllTests.sh loop finishes, then the HTML that was created is moved to our reports folder to function as a professional-grade report.

The report is still a work in progress, but it currently outputs the 5 tests we were conducting and uses Galaxy's color and layout scheme for the tests, which looks really nice (and will look better when we finish all the updates to it).

To conclude, one thing I want to get working is creating a soft link from our src directory (currently empty) to the location of our trunk (master branch) version of Galaxy since we have put it in a different place and our current relative paths are based upon the current structure. If this soft link is created, then anyone navigating the folder structure will be able to do so however they please.

Music listened to while blogging: The C90s & J. Cole

Reflection

For this post, I will be reflecting upon my recent Software Engineering test and doing a quick update.

The test shocked me at first, which it shouldn't have. It shocked me because I felt like I was writing essays upon essays, it being a complete short answer test. This shouldn't have shocked me, though, considering the nature of software engineering. One thing that did really mess with me on the test, however, was the way we had to recall answers. For example, one of the questions on the test asked about the dual-use dilemma; really just define and describe it. If I am correct now, then I do believe that the dual-use dilemma is that code that you produce can always be misused by someone else for malicious purposes and this is something to keep in mind when creating and disseminating code. This is an idea that is pretty secondhand to most computer scientists in my position, but knowing it specifically by the name 'dual-use dilemma' proved to be the only challenge for that problem. I guess this all really stems from my dislike of typical testing. I prefer a more hands-on approach to tests, rather than just written exams.

Overall, though, I do believe the test went rather well and did test some useful techniques within the realm of computer science and software engineering. For example, we had to detail test cases for a method that was given to us. Figuring out test cases is always a tricky thing to do because of the dozens of trivial examples that can arise that need to be tested. There's usually endpoint cases, blank cases, maximum cases, etc.

As for my update: I'm currently in the works of getting my SpotifyU internship off the ground. It's essentially a marketing internship, but it gives me a chance to work with Spotify, a program that I am, pretty much, obsessed with. I've been enrolled in the intern program for a while, but they have been pretty slow to get us the material we need to actually do the work. My personal research with Learn2Mine is going very well. Obviously I wish I could spend more time on it, much like I did in the Summer, but classes do get in the way of that. One of the reasons I am pumped for graduate school is so that I can spend more time doing research (50/50 ratio of class/research is my hope). That being said, I'm still waiting to hear back from a conference where we have submitted a paper to be a part of the conference proceedings at SIGCSE.

Music listened to while blogging: Badfinger (Breaking Bad inspired this) & Ellie Goulding