Working on a longer coursera project locally on Lubuntu

Today I decided to return to the practice of using a VCS due to the data science capstone project's growth. I have repos here and there, but somehow they just don't want to perfectly fit my needs (there's either size limit, lack of privacy or I'd have to pay).

I already believed I needed a local VCS since on longer tube journeys and flights I may not have internet access.

I supposed I'd choose git as I was hoping to deepen my experience with it, so that I can get more effective with the online hosted version as well - just about everything seems to happen on GitHub these days.

The client

I first checked out the clients - at least those available on Linux. I so far haven't come upon those feature rich UI's that I was used to on Windows, but I still demand something similar, so it was crucial (those tiny little annoyances which often go unnoticed, can really slow down work).

Formerly I didn't have much luck when choosing the weapons for Subversion, back then I couldn't find a *nix match for the handy TortoiseSVN. I have RapidSVN but the version I have is exhaustingly bogus.

As git I assumed is trendier, I had better hopes for better maintained clients. I found the default range on this page, and with my criteria in mind (free, open-source, working, preferably cross-platform) this quickly narrowed down to git-cola.

It appears good enough (although it froze a couple of times while authenticating with the server in my tests), however the aptitude version is way outdated (or at least its about dialog, pretending to be from 2012), so I had to uninstall that one (apt-get remove git-cola), and install it from source.

It is also written in python, which I'm learning as time allows, so the sources might give me a lesson or two on that.

The "server"

While some git components, especially the CLI clients should have already been present on my system, I checked the official installation steps and kicked off with a further

sudo apt-get install git-all

just to be sure ... further from that, after a little thinking and experimenting, it turns out that the client tools with a local repository are probably all I needed to get going with my local project.

... and it works!

I only had to initialize a new repo, and it was working straight ahead.
I mean, after untarring, it already works with

~/Downloads/git-cola-2.5/bin/git-cola

if someone enjoys inconvenience. I decided to call it a day - TBC.

-----

And as a little perk for the night, I accidentally (was queued for later in preference over some Netflix whatnot...) watched a TED talk with Linus Torvalds which surprised me with mentioning his share of creating Git as well as Linux. Nicely done!

Update

make prefix=/usr install

executed from within the git-cola directory (~/Downloads/git-cola-2.5 according to the above) does the job, so that git-cola can be now launched with ease from the terminal, and also gets available from the "Programming" menu group.

Sorry about the formatting, at this point I'm lazy.

GitHub and R

I've been to a workshop preceding a meetup where the basics of how GitHub can be conveniently used with R Studio was the topic.

One benefit: I finally got my hands a little dirty with creating GitHub issues (although to date I tried to avoid that ... partly because I only ever had public repositories and possibly I didn't want to share every everything, especially not exemplary beginner's mistakes :) ).

Another one: Being reminded of continuous integration. One day I'm sure it'll come useful. The only problem with it is it didn't fit into the workshop, but here are the related slides for that one day:

http://mangothecat.github.io/github-workshop/05-best-practices.html#continuous-integration

And, say, the third: Catching sight of the "git" tab in the R Studio GUI, once that's sufficiently enabled for the project. Again, one fine day ...

R: the window diff exercise

The implementation of moving/sliding window transformations is one of the basic duties one may come across while developing various monitoring, time series analysis or signal processing tools. Third party libraries probably embody solutions, but it's a classic so I'll devote an entry to it.

An example could be calculating moving average over a time series. This pattern popped its head up again at an R coding dojo (in about 2013 ... that's how fast I write :) ). The concrete scenario has something to do with the fingerprinting that some antivirus software perform in the heuristic search for high-entropy chunks in a data stream (file), which if found, suggests encryption was applied, possibly by a hiding virus (Derek's Shape Of Code entry has more about it).

Moving away from his original topic - where the brevity of R does a better job than a lengthier code, it appeared to me this could be a nice example to illustrate a performance optimization concept which can come handy at other times, certainly in some more performance-oriented scenario.

Notations

n: number of samples (say ~ 1M)
m: window size (say ~ 1k)
k: a numerical constant (the point is it is independent from the parameters)

The original approach

cost ~ n * k * m

Thinking of the aforementioned R code, that one browses through all possible positions for time windows of a certain size, and computes the ratio of uniquely occurring symbols to the size of the time window for each. The excerpt I'll focus on:

This calculation can become massive, as its cost (in that form) is proportional with the window size multiplied by roughly the number of window positions - sapply() executes for each relevant X, and calls unique() with a slice of the input data. The procedure examines most input symbols in the stream window.size times. Is it necessary? (Some relief is that the data is recalled from the CPU cache already, but with a little thinking we can come up with something better.)

Although R may not be the most efficient environment for running such code (as loops are preferred to be replaced with array operations, which could allow even for implicit parallelization), I will show how it is possible to reduce the complexity of the algorithm from somewhat an n * m to somewhat of an n + m (asymptotically) by applying a simple way of thinking twice (and I can tell that count.positives() accounts for the multiplier of "m").

Remark: actually, the original considers only each 5th one, but I'll take another route here, and so will ignore this.

First improvement

Assuming instead that we have knowledge of the first frame, it is possible to only take into account the bits that leave the window and those that move into it, namely the symbol that gets excluded on the left, and the one that gets included on the right. So, by focusing on the right items, the computational complexity (however, not in an asymptotic sense, yet) can be reduced.

cost ~ n * k2 * m + m * k3, main point is k2 < k (expected)

The frequency[] array holds occurrence counters inside the window. Once initialized, it is only maintained at a low(er) cost per window, hence the memory read operations become less redundant - although each sample is accessed twice.

Second improvement

Now one might go back in thinking to that the subject examined is basically the number of those symbol counters which are non-zeroes. When changing a counter it may either enter that set, or leave it (i.e. change to zero). Therefore, once we have a starting point (described the first window) it is sufficient to focus our attention solely on the values which significantly change - were positive and are becoming zeroes (meaning the related symbol is no longer present in the new window), or were zeroes and are becoming positive (meaning the related symbol appears in the new window).

So similarly:

cost = O(n + m), m for initialization, n for each step

This now has the promised characteristics.

As a note, when sliding a window over floating point values, e.g. for the moving average calculation, one should note that a value may have an impact of a different precision when entering and exiting the window. This may lead to accumulating numeric errors, and in the end, have a significant impact on the value computed for the last window position. A similar problem emerges if the data changes in the magnitude of variation over certain sufficiently lengthy intervals. But the above are quantized values - integers on each level of abstraction, so no rounding takes place, and this implementation caveat is naturally avoided.

At the finish line

The output of the code on my laptop is as below:

   Original  First imp. Second imp.
      5.846       3.443       2.051

Apparently, each iteration improved a little on the performance, over 2.5x in the end, although this number is not relevant. A true maximalist could go on for an rpp implementation, making it really fast.
Part of the reasons is that such an approach could further benefit from the unfair advantage of a C++ compiled parameter passing, which the first code cannot utilize (however, that one does have/tries to leverage the advantage of vector calculations to some extent).
Yet, the above is rather a demonstration of the concept than an über-optimization attempt, so no C++ comparison for now. The lucky thing (due to there are many constants tampering with the end results) is that even if the two improvements work with for loops, they prove faster in 'at least one' realistic run.

Closing words: this entry was partly written to exercise my writing skills, also as a follow up from years ago, however little significance it might be of, I liked creating it. The code somehow reminds me of code dojo exercises. (Not sure if this pattern could make it to be a Code Kata I heard about a good while ago?  One more thing I'll have to check out anyway, at least.)


To those who made it this far: thanks for reading!


The complete R code is available here.

R: why using require() over library() shouldn't be thy default

This is a very simple thing, and really isn't worth wasting too many words on it (but I will! :) ) The point I'm making is right there in the CRAN documentation on these functions:

"require is designed for use inside other functions; it returns FALSE and gives a warning (rather than an error as library() does by default) if the package does not exist."

The trend seems to be that people in increasing numbers started to prefer require(), however. In the one-off recipes, which are a relevant portion of what's being created with R, this doesn't make much sense.

For instance, at the beginning of one of XGBoost's demos, you'll find this:

require(xgboost)
require(Matrix)
require(data.table)
if (!require(vcd)) {
  install.packages('vcd') #Available in Cran. [...]
  require(vcd)
}

Created by Pretty R at inside-R.org

The first 3 require() calls do something. Then something else happens. Then some output, the main point of the demonstration is printed. The warning messages, possibly thrown in the header, will easily go unnoticed among the output lines, unless someone's keen to scroll up to the top.

So let's get together some reasons against the overuse of require():

#1 Delayed feedback. Fatal problems better turn out at an early point and an obvious way if it comes at no extra cost. Errors from require() may take extra effort to find.

#2 Risks. A system silently executing on with problems nobody expected, is always a jeopardy. It would be even more interesting with function name collisions - once the correct library() is missing, a standard function in the environment may get used instead of what the author hoped for.

#3 Copycats (education). Furthermore, the reader/user of a tutorial code does not necessarily have that much routine and is easily deceived for a while - and while that period lasts, this bogus practice will be copied all over the web, in Kaggle scripts and blog entries (like this one ;) ).

As a side note, in the mentioned code, the lucky thing is that the packages are later on used at an early point in time, so the problems will likely turn out at an early point anyway, data.table creation and similar things make this happen. But this is far from making the code correct.

GPU, Lubuntu, ML - bloody roots

I have presented myself with a CUDA-compatible VGA card for Christmas (Asus GT730) so that I can check out Theano and GPU computing in general. No matter the appetite, having the appropriate amount of business to deal with, I was conveniently procrastinating the installation. Could be good intuition :)

I tried reinstalling it a few times, until I found that some RAM-drive for whatever reason needs to be purged:

sudo update-initramfs -u

Running this at the beginning & also the end of the previously attempted installation steps, I believe it's "there" now, although it was missing a couple of things.

When running Octave, it was still complaining for not having a video driver in place

Xlib:  extension "GLX" missing on display ":0".

A page said that adding a couple of lines to the code (it was the Coursera Machine Learning course's ex3.m) can remedy the situation ... and in fact I only needed to insert the line

graphics_toolkit gnuplot

right after the initial clean-up:

%% Initialization
clear ; close all; clc

and that solved this one of my problems. Or I could at least see that initial plot.


Some of the Cuda examples are broken now, though. Not sure why (perhaps just a missing path), but when running

MonteCarloMultiGPU

I'm getting this:

error while loading shared libraries: libcurand.so.7.5: cannot open shared object file: No such file or directory

... thank you so much, laters, then ...

Samba to Optiplex

Looks like I have installed Samba on my LUbuntu desktop PC. While the server-side setup happened via a little bit of config editing (mainly of the smb.conf file in /etc/samba/), accessing/verifying the folders was much more straightforward.

On the server-side, one (to me interesting) caveat was having to, beyond setting read only = no, explicitly specify writable = yes. This took me a while to notice and I was quite unsuccessfully suspecting group access rights and other innocent objects around here. While this peculiarity could be down to some misterious mistake I made, and is contradicting some key equivalence mentioned on the web, it seems to work well now.



Not so much effort on the client-side: as mentioned here, pcmanfm allows URI's in the form smb://server_address to access the root folder.

Nice and easy.