Short Update

It's been a while since my last post because I have been busy.

A quick personal update:
1) I am now attending the Master's program at CofC and am taking 2 classes currently (CSIS 602 - Software Engineering & CSIS 604 - Distributed Systems)
2) I am now, officially, a Graduate Research Assistant doing research with CofC (with MUSC collaborators)
3) I also have a web development job for Innovative Resource Management
4) I have a publication coming out in the FIE (Frontiers in Education) 2014 journal [should be out around October]

So, for my research, I am having to learn (partially review) certain topics. I am choosing to use Python for my data munging and analysis for data I am dealing with for my research. Naturally, this is going to require the usage of the numpy and pandas libraries. I have used numpy in the past when building decision trees, naive bayes models, neural networks, etc. so I am mostly familiar with it and I am merely reviewing it. Pandas, however, is a library with which I have never dealt. To review, I am going through tutorials.

Music listened to while blogging: Bibio

NLP Paper Review

For this post, I will be sharing a Prezi presentation that I recently presented on a paper from the NLP conference I attended in Baltimore.

http://prezi.com/zac-hm_osqwo/?utm_campaign=share&utm_medium=copy

The presentation really speaks for itself.

Music listened to while blogging: GEMS

ACL 2014: BioNLP Conference

So I recently came back from the Association for Computational Linguistics', of which I am now a member, annual BioNLP (Biological Natural Language Processing) conference in Baltimore, Maryland. Firstly, I was in a hotel across from the Baltimore Orioles stadium and, not being the biggest fan of baseball, I definitely got my fill of baseball fans, hat-sellers, and hot dog vendors constantly yelling about their hate for the Yankees.

Since I only attended the 2 day workshop this was a little different than my normal conference travel. I attended sessions where people were doing molecular NLP tasks (such as querying Pubmed and other journals) in order to garner data and conduct metadata or real data analysis. These researchers typically utilized SVMs in their algorithmic analysis for their results, which gives me good ideas about where to take my own research. Unfortunately, most of the sessions were molecular NLP-oriented tasks, whereas my focus is more on clinical NLP, which is a different type of problem, by nature. Namely, scientific/structured writing is a lot easier to parse rather than unstructured notes written by different medical professionals.

No one at the conference is using the NLP system that I am using, which was a disappointment, but I was able to broaden my horizons to other systems such as i2b2 and biocreative. In my own research we are utilizing cTAKES/ytex.The conference included a panel of scientists that helped create these newer systems so that was a nice surprise to hear what it's like on the other side of research.

I will elaborate more on these systems when we decide if we want to steer away from the usage of cTAKES for one of these newer systems or if we decide to keep going down the road with which we are familiar.

Music listened to while blogging: Hellyeah

ACL 2014 and Journal Acceptance

So for this post, I will give a quick update on what I've been up to since the Summer started.

First, I am going to Baltimore tomorrow to attend a workshop 6/26-6/27 on Biomedical Natural Language Processing (BioNLP). The workshop, part of the Association for Computational Linguistics 2014 annual conference hosted by Johns Hopkins University, includes presentations on the creation of NLP techniques for parsing, the analysis of NLP-parsed data (specifically biomedical), and the utilization of tools/resources such as the Unified Medical Language Systems (UMLS), Systematized Nomenclature of Medicine (SNOMED) resources, among many others.

Recently, a paper submitted to the Frontiers In Education 2014 conference was accepted. I'll talk more about this conference and our paper that was submitted whenever it is time for the conference. I really hope I can attend this conference because it is in Madrid, Spain, which would be a great place to visit. Additionally, this conference would potentially help the Learn2Mine team to garner new ideas to incorporate into our own application.

Music listened to while blogging: Sublime

Data Mining with Weka

Another skill I'm going to have to re-pick up over the course of my graduate career is data mining with pre-built informatics systems. I can write as many data mining algorithms as I want, but preparing for the scalability required for big datasets is something I would not have too much fun doing.

So I can settle for using a pre-built system. For data analysis for my most current project, I will be using Weka.

Weka can be run from the command line or a GUI. The GUI is simplistic enough, though, to where the command line is not exactly the best way to go about these tasks, especially since a lot of the results of data mining algorithms rely on visualizations. For example, actually viewing a decision tree after creating it is pretty useful, rather than just reading information gains and prunes off of a screen.

Weka provides other built-in functionality which is immensely useful - the experimenter. So, a lot of times in data mining, you are faced with an issue where you don't know which algorithm will be the best after running it... and you don't know this for good reason. If you were an expert about the dataset and knew exactly how the features were interacting, then you could create an amazing hypothesis about which classifiers would run the best, but that is never a guaranteed solution. With Weka's experimenter, there is no need to worry about this. The experimenter allows you to select multiple algorithms to be run on a dataset (or datasets - a many to many relationship can be established). The results are then able to be viewed in a nice table format with sorting capabilities:

Above is the result of running the experimenter with three algorithms on Fisher's famous iris dataset. The ZeroR algorithm that was run is, essentially, a baseline as that is a classifier which does not rely on any of the features for prediction. So, effectively, the class label which appears the most in the training set will be selected for every datapoint which comes through to the testing set.

This may seem silly, but there does come a point where this is a crucial baseline. For example, if you have a dataset classifying whether or not someone has pancreatic cancer and your training set has 98% No's for the label (98% of the people in the training set do not have pancreatic cancer), then the test set will just say No for every single test. This can result in a high amount of false negatives, but... it will be correct a LOT of the time. So any classifier that you throw at that dataset will need to outperform 98% - this is really a situation where more data is needed, but being able to recognize that is vital to creating a good classifier.

The next 2 classifiers are J48 (weka's default decision tree algorithm) and naive bayes. These are both classic algorithms in data science so I will not dive too much into these for now.

Music listened to while blogging: Mac Miller

A Refreshing SQL

So becoming a part of a bigger project has induced the need for me to re-learn some old skills.

I have not written my own SQL queries since my DATA 210 class 2½ years ago so a crash-course was needed. I started by visiting the w3schools tutorial which worked great for me since I am re-learning as opposed to learning from scratch. Additionally, I needed to review regular expressions and how to actually do coding with them (I have done regex in the past by hand, but I never actually programmed them myself). I found a good reference site through Oracle.

Some of the main things that I took away were as follows:

• SQL Joins

Joins are probably one of the harder techniques for people to grasp when it comes to query languages. Inner joins are the most common form of joins (in SQL the JOIN command defaults to inner join). An inner join on 2 tables will result in a single table with data from both tables. The join requires one field from each of the tables to be equivalent as that would be the feature which is being joined "ON". So if you have a foreign key in a table, then you know that those two tables can be joined upon because of the constraints imposed upon foreign/primary keys. Outer joins differ from inner joins in that there will be a lot of empty data if an outer join is performed. There are different types of outer joins: left, right, and full. A left join will use the "left" table as the primary table being used while right corresponds to "right". So let's say we have a Persons table and an Amazon Orders table. We could easily have people in the Persons table that have not placed an Amazon Order. So the people in the Amazon Orders table will all be encapsulated by the Persons table. If we do a left join, then we will have one big table that has Persons listed with their Amazon Orders attributes added on to the end. For people that did not have Amazon Orders, there would be a NULL or empty value for those attributes.

• Altering Tables

Altering tables is a simple, yet vital functionality in database usage and administration. This can range from dropping tables or deleting attributes to designating new keys and features for a table. There is not much explanation needed to understand this.

• Populating Tables with the INSERT TO command

Creating new tables is insanely important when dealing with databases. Typically, though, you are not creating a table from scratch. You may want to take features from many tables, perhaps unstructured with a bunch of redundancies, and create a new, normalized table that prevents disk space waste and allows for easy querying. With the INSERT TO command you designate a table from which you want to grab information (as well as selecting the specific attributes you desire, if not all of them) and select a new table for which to send the information. 

Music listened to while blogging: Schoolboy Q

I'm going to start linking the artist's name directly to a song of theirs, to add a little interactivity to my music updates.

A Personal Hurrah and the Road Ahead

So I'm using this post to explain my hiatus from blogging.

I just recently graduated (May 10th) from The College of Charleston with B.S.'s in Data Science (cognate, an effective minor, in molecular biology) and Computer Science with a minor in mathematics.

I am now pursuing the Master's program at The College of Charleston for 2 years. This is the Computer and Information Sciences program - the concentration I am following is the Computer Science track, which is more of a classical theory-based approach to Computer Science. While doing this, I will be conducting research, hopefully as a GRA, graduate research assistant.

Currently, I am doing research for the summer between these programs with Dr. Anderson and others working with building and modifying machine learning algorithms. A lot of the specifics cannot really be elaborated upon at this point in time, but hopefully I will be able to disperse that information at some point because this work is going to be great and is quite the departure from my normal research routines. We're just waiting on IRB approval so I can get going with the data for the project.

Also, I have recently showed an undergraduate student the basic development environment for working with Learn2Mine. I am still one of the lead developers on the project, but a lot of the maintenance will slowly be transitioned to this undergraduate student as he becomes comfortable with the system.

Capstone: Upcoming Presentation

This post will be brief as it will be outlining my upcoming poster presentation for the Data Science and Computing in the Arts symposium at the College of Charleston, as well as my Capstone technical report.

A look at my poster:

I chose to go with a brief overview on the poster and I can elaborate more if people ask. So the poster does not go too much into the technical detail.

Below is a flier advertising the presentation:

RMH Homebase - Chapter 7 of Software Development: An Open Source Approach

This post will be composed of mostly responding to exercises found in Chapter 7 of Software Development: An Open Source Approach.

Chapter 7 is about the development of database modules and the chapter uses RMH Homebase (as I have referenced in numerous posts prior to this) as the example with which to conduct exercises.

The first exercise relates to database normalization criteria.

First, I would like to start by outlining the six database normalization criteria (directly taken from the text):

1. The rows can be rearranged without changing the meaning of the table (i.e., there's no implicit ordering or functional interdependency among the rows).
2. The columns can be rearranged without changing the meaning of the table (i.e., there's no implicit ordering or functional interdependency among the columns).
3. No two rows of a table are identical.  This is often accomplished by defining one column whose values are mutually unique.  This column is known as the table's primary key.
4. No row has any hidden components, such as an object id or a timestamp.
5. Every entry in the table has exactly one value of the appropriate type.
6. No attribute in the table is redundant with (i.e., appears as an explicit substring of) the primary key.

It is given that certain tables in RMH Homebase violate criteria 5 and 6. It is given that dbDates does not satisfy either of these criteria. 

Another table that violates criteria 5 is the dbSchedules table. To recapitulate, criteria 5 states that every entry has exactly 1 value; dbSchedules sometimes misuses the Persons field. Sometimes there only exists 1 person in the field (or null), but there do exist times where multiple people are in the field for one record. Because it is happening for one record, that is a violation of criteria 5.

Another table that violates criteria 6 is in the same exact table. Typically, databases will have a primary key in the form of some unique id. Other times, however, compound keys are used (or created). A compound key is the usage of multiple fields in a database table to uniquely identify a record. For example, if we had a Person table, then we could potentially have 2 people with the same name so maybe we would identify them by their name and address together. The dbSchedules table, though, uses name and phone number as a compound key, which is not a problem, but the primary key is now a new field. So the primary key (or compound key) is now a redundancy of both the name and phone number in the same table, which is in clear violation of criteria 6. 

The next exercise is asking for me to develop and unit test specific functions for the dbShifts.php module.

So the getters can be written very easily since id is a compounding of all the other needed attributes (delimited by -'s).

function get_shift_ABC($key) {
    $attribute = explode('-', $key);     $ABC = $attribute[fieldNum];
    return $ABC;        

}

So here I am showing a very generic version of the getters I would use, given the exercise specifications. I change the variable coming to "$key" from "$id" merely because I like key as a better name, but that is just my personal preference. The next line is equivalent to conducting a ".split()" on a string in Python. So now "$attribute" is a list of all the attributes, in generic list order. Now you may have noticed that I named the function ..._ABC(...) and named a variable $ABC. This is because each getter would have a better name for the variable there (for readability). For example, you would replace "ABC" with "month" if you were writing the get_shift_month(...) function. The only thing that changes when getting different fields with these getters is the indexing into the $attribute variable. Below I have listed what field each value in the list corresponds to:

fieldNum	Field
0	Month
1	Day
2	Year
3	Start
4	End

So now all 5 getters are, effectively, written.

The last exercise wants me to design and implement the changes to the database modules required by the new feature - Item 4: Calendar Month View - in the "wish list" as prescribed in Appendix B. The quickest way to do this is to copy an entire database through php and then use refactoring (this may not be the cleanest, but it is quick, effective, and gets the job done). The php class I refactored was dbWeeks - I chose dbWeeks because the two classes are organized similarly (this is even hinted at doing in the book - page 194). Refactoring allows me to change the fields to the necessary values. So now each row in this calendar month view represents a month, active or archived. The unit tests were also able to refactored easily and all the tests passed with no apparent issues.

Penultimately, I would like to give an update about Team Rocket and our work with Galaxy this semester. We have finished our poster and got it printed off. Below is a (low quality) picture of our poster that we will be presenting at the College of Charleston School of Science and Math Poster Session on Thursday, April 17, 2014. 

Lastly, I would like to give an update about my plans to "Meet Charleston" - for me that was attending an "Agile User Group" meeting. The next meeting will be taking place April 24, 2014, from 11:00am-1:00pm and will be hosted by Life Cycle Engineering. I am really excited to make it to this meeting.

Software Development: An Open Source Approach <-> RMH Homebase (Developing the Domain Classes)

For this post I will be going through Chapter 6 of Software Development: An Open Source Approach and reflecting upon my experience with some of the exercises at the end of the chapter. All of the exercises focus on dealing with the open source software RMH Homebase (which I have mentioned in posts in the past).

The previous blog post in which I talked about the installation and usage of RMH Homebase can be found here.

The RMH Homebase release 1.5 code base can be downloaded from myopensoftware.org/textbook

Unfortunately, all the work I did on my previous post is for naught for the purposes of doing these exercises since I am on a different computer. So let's run through some commands and dependencies with which I had to deal:

% sudo apt-get install mysql-client mysql-server
This command didn't work as I had hoped because it kept complaining about out-of-date dependencies and missing dependencies. And these errors kept continuing.

% sudo apt-get install mysql-server-5.5 mysql-client-5.5
led to
% sudo apt-get install libdbd-mysql-perl
led to
% libmysqlclient18
which ultimately led to an error with glusterfs-server, which is something, I believe, extremely specific to my lab computer.

Experience-wise, it is good that I can note what is going on. Practicality-wise, this is frustrating because I do not want to tinker with the cluster as there are a lot of projects that are relying on the cluster not being tinkered with (including my own). So while this was valuable, I will continue this on a different computer at a later point. Also, the gluster error perpetuated for every package I wanted to install (apache, php, etc.)

More commands:
% sudo apt-get install apache2
% sudo apt-get install php5
% sudo apt-get install phpmyadmin

For reference, an easier way to get all of the MySQL stuff installed can be done through:
% sudo apt-get install mysql-client-core-5.5 mysql-server-core-5.5

Everything that these commands install and do can be found in my older blog post I mentioned earlier.

The exercises are very basic, in nature. Defining new functions to set and retrieve the value of the variables $employer, $contact_person, and $contact_phone.

So, pretty much. Encapsulate the Person class with getters and setters so someone can say "get_employer()" or "set_employer($newEmployer)". So there's not really any work here rather than making getters and setters for all 3 of the features.

Example:
function set_employer($newEmployer){
     $this->employer = $newEmployer;
}
function get_employer(){
    return $this->employer;
}

The next step is to create a more well-defined constructor. This goes as follows:

function __construct($f, $1, $a, $c, $s, $z, $p1, $p2, $e, $t, $status, $employer, $contact, $contact_phone){
     $this->first_name = $f;
     $this->last_name = $l;
     .
     .
     .
}

It's obvious that you would just typical assignment in this constructor like I started to outline. When doing the password field you would probably want to throw in your favorite flavor of encryption so you are not handling raw passwords because that's a security no-no.

The next question pretty much asks you to rewrite the set_status function because currently the $value that is passed in is never checked to be valid (which it can be "active" or "inactive"). Currently, if something else is passed in (let's say "cheese") then someone's status could be set to "cheese" rather than actually checking if there is a problem. So let's rewrite the function below:

function set_status($value){
     if ($value == "active" or $value == "inactive"){
          $this->status = $value
     }
     else{
          echo "Your input for set_status was invalid. active and inactive are the only valid options."
     }
}

So now the user knows if they picked a wrong value for set_status and status does not get changed if an invalid input is put in.

The last exercise is to refactor the Person class (where we've been working this entire time) to have removed all the mutators that aren't called from anywhere in the code base. This can be done by deleting or commenting out (commenting out would be preferable in my eyes) unused mutators.

Music listened to while blogging: Childish Gambino