Deliverable #3 Experience

I'd like to start this post by showcasing the logo I made for my software engineering team, the Boolean Bombers, using Logo Garden. The logo can be seen on the right.

There were a few criticisms of our current testing suite for Galaxy. These were very quick, very easy fixes. I took the liberty to make the corrections.

For starters, we were creating a Python file for each test case, which is not what we should have been doing. What is funny about this, though, is that when we were creating these python files, the only difference was the filename that we were using to find the testCaseX.txt file. So it was very easy to create a loop that runs all the code for each testCaseX.txt file within the testCases/ directory. This avoids storing loops and works regardless of how many or how few test cases we have.

Additionally, I added a quick boolean to check if we even had a test case to begin with. If we didn't have a test case, then we jump into an if statement at the end of the driver that reports to the html that there were no tests.

On top of all this, I added a quick line of code to the runAllTests.sh script that opens the browser with test results and, if a browser is already open, then it opens the results in a new tab in that browser.

To add to that, I made some quick changes in the CSS in the HTMLBackbone that retains styling to keep the namesake for testing (pass vs fail, rather than success vs fail/error).

All things considered, we are in excellent shape for the rest of this project.

Aside: I am a part of a Spotify internship right now and one of the things we have to do is try and refer people to Spotify. There's a contest going on right now where we can earn rewards based on who gets the most people referred. My referral URL is here. 

Music listened to while blogging: Donald Glover comedy album ~ Weirdo

Galaxy Deliverable #3

For this post, I will be reflecting upon the final edits of my team's Galaxy Testing Project before our next deadline (tomorrow).

For starters, the testing framework did not remove files from the temp directory. The temp directory held all the reports that are produced by tests (1 report per test) that are then compared to the oracles. The framework now does this through the following call at the beginning of runAllTests.sh:

rm ../temp/*.*

The reason we do it this way rather than doing a recursive remove of the temp directory is so that we preserve the existence of the temp directory. We very easily could have taken an alternate route where we do the following:

rm -r ../temp

mkdir ../temp

These two segments of code achieve the same goal in the end.

Lastly, the testCaseX.txt files were cleaned up. Before, the "Expected Outcome:" section contained the name of the oracle file and the contents of the oracle file. This was a bit too redundant for my taste, so I removed the contents of the expected outcome from the testCaseX.txt file. The Python code did not have to be edited here since we checked the oracle for the actual results anyways. 

What we did change in this part of the framework, though, was the way in which oracles were found. Before, the validation of test results through the oracle was performed through the invocation of a method I created that took a test case number and then created a call to an oracle. Instead, the testCaseX.txt file is read and the actual oracle filename is read and then sought out by the validation method.

Music listened to while blogging: N/A

Learn2Mine and Skill Trees

This post will not be reflective of my team's ongoing Galaxy Testing project as we have not met since my last post (we will be meeting and working tomorrow morning). So, instead, I will detail my latest work on my own research in the Anderson Lab here at the College of Charleston.

Learn2Mine utilizes gamification in order to help teach students data science. My latest work on the application is on revamping the way in which users view their skill tree.

Fig. 1 - Old Skill Tree

The best way to talk about these concepts is to use the physical images. The initial skill tree implementation is able to be seen in figure 1. This skill tree used a regular block method to represent skills that users have unlocked, learned, or mastered. This method seems a bit too cut and dry, though. For starters, we had 2 skill trees with this current implementation to represent pattern recognition techniques in one tree and R programming skills in a different, second tree. Also, if a skill was 'locked', then users were not able to work with that skill.

Fig. 3 - Classification Subtree

Fig. 2 - New Skill Tree Prototype

Now we have departed from that. Figure 2 shows the entirety of our current skill tree prototype - this prototype is created through a javascript implementation of Cytoscape (a flash-based graphical environment). We give a  hierarchical progression of skills that are represented in tree-form. This is not strict, though; this hierarchy serves more as a guideline for what order we feel users will learn the best. There are many different progressions one can take through the skill tree. For example, you can go down the classification branch in the tree (as seen in figure 3). This will have you learning the basics of the K-Nearest Neighbor algorithm, Partial Least Squares Regression, and Neural Networks. We also have a mastery lesson for K-Nearest Neighbors that requires users to implement optimization techniques in order to improve their KNN classification over a specific threshold.

We make new lessons just about every week for Learn2Mine, as it is going hand-in-hand with Dr Anderson's Data Science 101 class. This is functioning as a replacement for other workflow management systems that implement data science techniques. Typical systems used in DATA 101 include Weka and RapidMiner - these systems have lots of flaws, flaws that we are trying to combat with our system, while also focusing on the educational aspect of this process. This launch in the DATA 101 class is also functioning as our pilot test deployment of the system.

Music listened to while blogging: 50 Cent and J. Cole

Further Information on Unit Testing with Galaxy

There has not been much headway with the Galaxy testing project since my last blog post, but we did run into some hurdles we did not foresee.

Our original implementation of our unit testing architecture banked on the usage of datasets. This seemed natural since Galaxy always uses datasets when doing its testing. The reason for this is because Galaxy's primary use is for dealing with large biological datasets, which naturally lend themselves to dataset files, rather than manual input. Since we are testing such small pieces of the architecture and tools, it may be more helpful for us to input actual values rather than forcing a dataset constraint.

In Python, this is extremely easy. Within our testCaseX.txt file, we just need to specify some kind of delimiter or boolean that can be programmatically read. The presence (or absence) of this value could mean just read the raw data value out of the testCaseX.txt file, rather than finding a dataset in a different folder. This will allow for ease when adding future tests. This also adds an extra way to break tests and see if they fail and break down as expected.

This, and much more, should be implemented before my next update here and I will reflect on those changes, along with the associated obstacles, in my next post. Consider all things, this project seems to be moving along smoothly, even though the team dynamic feels a bit lacking.

Music listened to while blogging: Kendrick Lamar and The C90s

Writing Unit Tests for Galaxy

In this post I will be examining and reflecting the processes which must be undergone in order to write a test case for Galaxy, from beginning to end.

The basic rundown of the calling hierarchy:

1. sh runAllTests.sh
2. testCaseX.py (testCasesExecutables/)
3. testCaseX.txt (testCases/)  
4. testCaseDatasets/
5. testCaseXReport.txt (temp/) and testCaseXOracle.txt (oracles/)
6. Team3_Testing_Report.html (reports/)

1. Running all the test cases
There is a script located in the scripts/ folder known as runAllTests.sh. This script first removes the last testing report that has been produced. In the future, this will be modified to also delete test case reports, but more on that later. Second, the script iterates through every file in the testCasesExecutables/ folder and runs them using a for-each loop with a python call. It is worth noting that this also runs our helper file, but, because it is only a declaration of functions, it does not interfere with any calls or act as a detriment to timing for these tests. So within each call of a test case executable file we have to consider how we will be...

2. Running the executable
The first executable that will be called is testCase1.py. This file looks very clean as we have abstracted a lot of work that would be repeated in every test case file. We make the imports to subprocess, os, and testCaseFunctions. Subprocess is a module that we can import to make command line calls, a pretty vital element to testing from the command line. The invocation using subprocess is seen toward the end of the executable where we use the line:

# Let's make the call to the shell
subprocess.call(testCaseScript,shell=True)

This allows us to make a call directly to the shell - this is a security issue, but this is a project of ours we are running on open-source software, so this is not too much of a worry for us.

We make a call to os in order to have a default directory we are making calls from - this allows us to write relative paths in a more readable manner. This is done with the following line: 

# Change the working directory to Galaxy's Home section to make our calls simpler (this also enforces our helper method calls)
os.chdir('../project/src/galaxy-dist/')

Lastly, we call testCaseFunctions in most other lines of the file. This is to interpret the textual version of the test case, to assemble our command line call, validate results with our testing oracles, and, finally, produce a professional-grade testing report.

3. Textual test case

Located within the testCases/ folder, our textual test cases are a vital part of guiding our test cases. For explanation, I will be referring to testCase1.txt. There exists full documentation about this file within the README within the Docs/ folder of our project, but I will detail a very basic version of the rundown. 

Test ID:

A test ID is a textual description of the test being managed at that point. This ends up being passed to the final html. This ID is unique because no test case should be the same as a test case that already exists (but there are not preventative measures stopping someone from adding a congruous test case).

Requirement Being Tested:

This section names the function we are dealing with (for HTML purposes) along with the atomic requirement being handled. In our example, this specific requirement is: "Calculate the length of a single fasta-encoded string. Length = 1."

Component Being Tested:

Galaxy's hierarchy very easily allows us to pick out what component is being tested with this specific requirement. Galaxy has toolsets which are made up of various tools. This tools are, typically, one function tools (in this example, "fasta_compute_length" is one tool that performs one very basic function within the "fasta_tools" toolset). This allows us to pick out the name being managed and even use it when building relative directories into our test cases.

Method Being Tested:

This details the actual method that we are handling. In our example, the entire tool is one method and it is the one we are testing. 

Test Inputs:

Test inputs make up the instantiation of the method we are testing. There is a line for each set of inputs we would need in order to test a tool and this is all handled with Python scripting. The order of the inputs is very important, though, as we have to build command-line arguments with these inputs and misordering them can confuse the command line and produce errors or false results.

Expected Outcome:

Expected outcome details which oracle we should refer to whenever comparing our testing report. Additionally, there is a textual description of what we expect to see under the oracle file.

4. Test case datasets

The nature of Galaxy lends itself to using full-on datasets rather than small inputs. Galaxy is meant to be use for data-intensive biology, but there is nothing stopping us from using the methods and input types of large biology datasets with our own test cases. In fact, this is the only way to effectively conduct tests of this nature. For example, within our testCaseDatasets/ folder, we have, currently, 5 FASTA files that contain data that is interpreted by the individual test cases. Following with the example we were using earlier, the testCase1.fasta file is a very simple file that contains a FASTA ID (no restrictions other than having to start with a ">") and then an encoded string on the very next line. The test case for the first test is about measuring a FASTA string that has only one character, hence only the value "C" being in the string section of that FASTA file.

5. Textual test case reports

Each individual test that is executed creates a textual report within the temp/ directory. Currently, each time a test is run, the new test case report will overwrite the old test case report. This is effective and it works, but we are going to work on getting the runAllTests.sh file to also remove all files within this directory (which should be easy with a rm -R call). These reports are nothing more than the results of running the tool with our given inputs. These files are what are compared to the oracle files. The files have to be exactly the same or else there will be a result that ends with a failed test.

6. Full testing report

While we compare each of those textual test cases to their oracles, we are slowly building up an HTML report for a final result within our reports/ section. This is done by using a htmlBackbone.html file we have obtained and appending individual testing results to our full report. The backbone came from running one of Galaxy's built-in functional test and then we stripped out all of the nonsense that was not CSS. This way, we can use Galaxy's CSS while using our tests and create a professional-grade report that mirrors the look and feel of Galaxy.

Music listened to while blogging: Spotify radio station based off of C90s - Shine a Light (Flight Facilities Remix)

Deliverable 2 Meta-Experience

From my title you should be able to deduce that the bulk of this blog post will be about the experience I, and my team, had whenever we presented our second deliverable for our testing project.

What I thought was going to be a glorious experience turned out to be one of the more horrific experiences I have had. I thought I had set my team up to be pretty far ahead in our project by already coding up our test cases and having HTML output. Apparently, though, the way I was conducting the tests did not follow the exact specifications for my Software Engineering class. I incorporated a couple of the file calls from Galaxy into my test cases and apparently that is not allowed. We have to, instead, reinvent the wheel. I was improving upon Galaxy's testing suite by implementing a way to conduct individual tests from individual tools (something they have listed in their documentation and wiki as something they need to do). Instead, though, we have to develop a testing suite that stands alone from the Galaxy project. The way I was doing it would be great to have submitted to Galaxy, but it would require too much work to do that and fit the specifications for the Software Engineering team project. While this is disappointing, this is the work that I have been given and, much like in the real-world when working for a company, the work given by the customer is the work that needs to be done.

All in all, this has been a good learning experience, though frustratingly wasteful for me when considering the amount of time I dumped into the functional testing.

The next step for my group and me is to examine the individual units that underlie the functions we have been examining. We have been examining the "Upload Data" toolset and this toolset has a multitude of helper functions that it requires in order to run. By extracting these helper functions, we can test each of them individually which, in essence, is a parallelization of unit testing as this goes hand-in-hand with it. If we can establish unit tests that work on these helper functions, then we may be able to work our way all the way up to testing a function that utilizes these test cases, which, in essence, is creating a functional test. In my eyes, this would be an immense, intensive learning experience that everyone in my team can benefit from.

Music listened to while blogging: The C90s

Deliverable 2 Experience

For starters, this second deliverable tripped my team and me up quite a bit.

Initially, I thought we had to implement 5 test cases (not just identify them) so I took the leap and created those test cases and the HTML output to follow. This is a requirement for the third deliverable of this testing project so at least I am ahead of the game on that.

Subversion, however, has decided that we have to become mortal enemies. When I initially downloaded our team repository, I ended up pulling each of my team's individual development branches in addition to the trunk, which we tried to avoid. We did this so we can simply do merges when we were confident with our changes. We wanted to test our changes and tests before merging our branches with the head of the master branch.

When I tried to push my final testing scripts into Subversion my GUI, RapidSVN, keeps mentioning "Unknown Error!" and a 405 error, which they really do not detail much. Upon short searching, I found that some people were able to fix this problem by re-checking out the repository and then proceeding to make changes. I am currently doing this, but Subversion likes to take its time and take hours to pull down development branches so I am currently waiting on that to finish.
Update: I just made it so I only pulled out the folder I need from the repository and Subversion finished the checkout in a matter of minutes. I then committed my changes up and they can now be viewed in the team repository (link on the right side of the page).

My goal is to get these scripts up before our presentation of our second deliverable tomorrow. I created python files that make a BASH call to a built-in Galaxy method that allows the running of functional tests. I decided to make the call this way because Galaxy creates a minimal HTML file with some important data about the test. So we developed tests and pass them to this BASH call.

The overarching structure:

runAllTests.sh loops and makes calls to each of the 5 test cases
The first test case generates the initial html file after the test finishes running
The proceeding test cases append to that html when their respective tests finish (Using Python to generate HTML code)
Once the runAllTests.sh loop finishes, then the HTML that was created is moved to our reports folder to function as a professional-grade report.

The report is still a work in progress, but it currently outputs the 5 tests we were conducting and uses Galaxy's color and layout scheme for the tests, which looks really nice (and will look better when we finish all the updates to it).

To conclude, one thing I want to get working is creating a soft link from our src directory (currently empty) to the location of our trunk (master branch) version of Galaxy since we have put it in a different place and our current relative paths are based upon the current structure. If this soft link is created, then anyone navigating the folder structure will be able to do so however they please.

Music listened to while blogging: The C90s & J. Cole

Reflection

For this post, I will be reflecting upon my recent Software Engineering test and doing a quick update.

The test shocked me at first, which it shouldn't have. It shocked me because I felt like I was writing essays upon essays, it being a complete short answer test. This shouldn't have shocked me, though, considering the nature of software engineering. One thing that did really mess with me on the test, however, was the way we had to recall answers. For example, one of the questions on the test asked about the dual-use dilemma; really just define and describe it. If I am correct now, then I do believe that the dual-use dilemma is that code that you produce can always be misused by someone else for malicious purposes and this is something to keep in mind when creating and disseminating code. This is an idea that is pretty secondhand to most computer scientists in my position, but knowing it specifically by the name 'dual-use dilemma' proved to be the only challenge for that problem. I guess this all really stems from my dislike of typical testing. I prefer a more hands-on approach to tests, rather than just written exams.

Overall, though, I do believe the test went rather well and did test some useful techniques within the realm of computer science and software engineering. For example, we had to detail test cases for a method that was given to us. Figuring out test cases is always a tricky thing to do because of the dozens of trivial examples that can arise that need to be tested. There's usually endpoint cases, blank cases, maximum cases, etc.

As for my update: I'm currently in the works of getting my SpotifyU internship off the ground. It's essentially a marketing internship, but it gives me a chance to work with Spotify, a program that I am, pretty much, obsessed with. I've been enrolled in the intern program for a while, but they have been pretty slow to get us the material we need to actually do the work. My personal research with Learn2Mine is going very well. Obviously I wish I could spend more time on it, much like I did in the Summer, but classes do get in the way of that. One of the reasons I am pumped for graduate school is so that I can spend more time doing research (50/50 ratio of class/research is my hope). That being said, I'm still waiting to hear back from a conference where we have submitted a paper to be a part of the conference proceedings at SIGCSE.

Music listened to while blogging: Badfinger (Breaking Bad inspired this) & Ellie Goulding

General Work and Reflection

This post will be unlike a lot of the posts I've had in the past. I will be reflecting on work from most of my classes, rather than just Software Engineering. This is because there have not been any major things done to our Galaxy project since my previous blog posting.

For the Galaxy project, we presented our findings and results of our first deliverable to the class. This included an overview of the project, how we ran the built-in tests, and our experience. Galaxy has built-in python files that can be run in order to conduct tests. One example is "run_functional_tests.py". This runs all the functional tests (takes hours to run) built into Galaxy. Optionally, you can specify a parameter that allows you to run a specific subset of functional tests. We will definitely be doing that whenever going through our tests cases because it would be too much of a hassle to have to wait 6 hours for each time we want to run through our testing suite.

In some of my other classes, we have been working on various projects. In Bioinformatics, I am currently working on a partnered assignment where we have to find the longest common subsequence in two strands of DNA. It's odd because our code, in Python, works on small versions of the problem and it works for the examples given, but our code does not produce the correct answer when we have to submit it for a larger problem. It's very unlikely that the site we are submitting to has errors, but I'm starting to question it at this point. In Advanced Algorithms, we just received back tests whose main subject was computational complexity of various algorithms and applied analysis of other algorithms; we looked at topics such as searching and sorting algorithms, big O, big Θ, and big Ω, recurrence relations, and other topics. In Programming Language Concepts, we have been studying regular expressions, grammars, compilers (every part: Scanner -> Parser -> ...), C, and pointers. In Public Speaking, I am currently preparing a speech that has the objective of informing the audience about the marvels of AI and how it will affect everyone in the future. Aside from that, work on my Bachelor's Essay has not gone into full swing yet as we are still in a heavy development phase for Learn2Mine. We are in the process of beefing up the security for the site as there is an exploit that we found that we are vigorously working to patch up.

First Deliverable Experience (Galaxy)

Working on Galaxy always turns out to be an interesting experience. Getting it installed on my virtual machine was pretty seamless once I figured out which dependencies were needed. Python was pre-installed with Linux Mint 15 so I did not need to worry about getting a compatible version of Python.

Unfortunately, we, my team and I, do not have access to our team SVN repository yet so we have been working in separate areas, still checking out files through SVN. I used my personal directory mentioned in an earlier blog post (./playground/Turner/galaxy/galaxy-dist/...)*.

Mercurial was needed to help initialize the Galaxy files. This was initialized very simply using the command:

sudo apt-get install mercurial

Mercurial is able to be called using the "hg" command (hg is the chemical symbol for Mercury - nifty little easter egg).

Once Mercurial is installed, you can run Galaxy. The source code does not need Mercurial to be extracted (or built), but it needs it for actually running the program itself. To initialize Galaxy, you navigate inside of the /galaxy-dist/ folder and use the command:

sh run.sh

This runs a bash script on the local machine (hence the "sh" for shell) and there exists a file within the /galaxy-dist/ folder named "run.sh". This file contains bash commands that start the Galaxy server.

This initiates Galaxy and it can then be redirected to by typing "localhost:8080" into a browser. Perhaps you have something else using your 8080 port, though. In order to modify the port on which Galaxy runs you would go into the "universe_wsgi.ini" file that exists in the /galaxy-dist/ folder. There are lines in this file that read as follows:

# The port on which to listen.
port = 8080

If you change this to 8081, for example, then you would navigate to "localhost:8081" in order to find the Galaxy interface. In order to push these changes into Galaxy, you have to stop the Galaxy instance from running (stopping the sh run.sh command). This can be done by using a keyboard interrupt (ctrl+c typically) in the terminal that started Galaxy. If, for some reason, Galaxy was initialized using the & (runs a file in the background so a keyboard interrupt is not possible), then you have to find the process number in order to stop Galaxy.

This can be done through the following process:

top

Using the "top" command allows you to view all the processes currently running on your machine. Galaxy is instated through a Python process. Finding the process number for a large Python process is the key here. Once found, make note of the number. Then the following command can be run to stop Galaxy:

sudo kill -9 $processnumber

where $processnumber is the process number of the Python process. Typically, kill -9 is frowned upon because it can lead to really bad cleanup on the computer-end. My simple solution to this is to not run Galaxy in the background. Just have it in a place where you can easily stop it. An alternative is to issue the restart command to Galaxy. Navigating to the /galaxy-dist/ directory and running:

sh run.sh --reload

This allows Galaxy to reload (restart).

All things considered, I think my team is really starting to understand Galaxy and I believe this testing project is going to go super-smoothly.

Music listened to while blogging: alt-j

*Update: We received access to our team repository as I was writing this blog so this will be updated accordingly.
The repository is now located at: https://svn.cs.cofc.edu/repos/CSCI362201302/team3/
Really, the only difference here is that instead of using /playground/Turner/ as the base of operations, we will be using /team3/ as the base of operations.