Processing close

People sometimes write code that is very assertive about this or that piece of a system. In a more suspicious case, many parts.
This typically isn't going to yield performant code - something will often try to slow your job down.

A specific advise:

0) Keep (simple) processing close to the database.

What I mean by simple is something that easily could be translated to a few machine level instructions (e.g.: sum) in each iteration, is done in a few iterations (e.g. 1), doesn't operate on a large dataset in each iteration.

Let's generalize it:

1) Keep avoiding to cross boundaries between system parts unnecessarily.

Or: try to consider alternatives where fewer communication routes are involved.

Such means of system boundary crossing to be avoid could be anything really, depending on the scale of operation, and at what magnification we consider the system:

• Querying data from a server
• Calling into a DLL/library
• Exchanging information with a service/daemon (inter-process - but also, probably with a smaller overhead, synchronized inter-thread communication).
• Looking up data from another table (one implication: look for unnecessary JOINs, maybe consider denormalization, esp. in a NoSQL setting)
• Calling Java bytecode from a compiled binary
• Calling another function (remember there's a cost of leaving return information on top of the stack)
• Blocking on-screen confirmation with the user

 

A major exception: if the overall complexity is high.

In this case it may be worth turning to some hard/software dedicated to the processing - using a GPU, an SSD, scaling up, a cluster etc. a few terms to think of as opposed to CPU/FPU, HDD, your regular hardware, single compute node. And this will all involve moving data from one part of the system, less suited for the processing task, to another, more feasible, possibly more dedicated, potentially a temporarily allocated resource from a cloud.

However, it's also worth noting:

 

faulty and or critical system components may cry for redundancy

Such as - people. And then there come the desirable boundaries - but also come peer insight. Whether the superior efficiency of the one-man teams is a myth... well, while we'd like to believe in myths, we do know, that may not always be the best that can happen. You may need very proficient and disciplined people for that to work out - these days, with an expanding IT industry, years of experience on average is going plummeting.
Stumbling upon them could be way more the exception than the norm.

A quick reminder to self about error handling in R (baby steps #1)

I guess the below code tells pretty much it all:

tryCatch({
  x = function() {
    stop("hahaha now you blew the code!")
  }
  y = function() {
    x()
  }
  y()
}, error=function(e){
  print(e$call)
  print(e$message)
})

And then sourcing it gives:

x()
[1] "hahaha now you blew the code!"

I guess that's all that there is in practice...
... except maybe that there's a finally parameter included which I never notice :

function (expr, ..., finally) 

So... worth a second look :)

(To be continued...)

JavaScript versus integer sequence

Looks like the ever feared, popularly hated JS really has its drawbacks ... you need functional programming to create an integer sequence? you need a loop? and wouldn't anybody want to just do something about it? :)

E.g.
https://stackoverflow.com/questions/3895478/does-javascript-have-a-method-like-range-to-generate-a-range-within-the-supp

Jeez :) time to realize how good Python and R are... well, R as a language only in certain aspects, but still.

Gesundheit MongoDB

MongoDB 3.6

Looking for null values? The answer:  BSON type 10.
You are to look for "$type: 10" values...


Naturally the question is whether you guys are seriously using a magic number by design in a database engine built for web developers, in the 21st century. Exciting times.

(What am I overlooking ...?)

Beauties of R: isNumeric

So, in R apparently there is no function in the core packages to test if a string would work out as a number.

There are really unappealing workarounds, like:
https://stackoverflow.com/questions/16194212/how-to-suppress-warnings-globally-in-an-r-script
This uses global settings (options(warn=-1), bonus points for the magic number), risking failing to recover in case of erroneous situations, potentially affecting other threads unwittingly/unintentionally as part of a larger scale quest, etc.


Probably the slightly more laycoder-minded need to lay out their homemade witchery for this maybe with a regex. Or use something slightly external e.g. limma from the bioconductor repository. (And then ... there is a function called isNumeric. How ... why wouldn't you subconsciously confuse that with is.numeric? Cautious language design could come to the rescue ...)

Seriously? Is it the time when non-mathsy JS's isNaN() functionality clearly beats R?? :D No way. What am I missing?

Anyway, just for the huge fun here's my quick regex solution...

> test.values = c("5", "56", "5.67", ".67e+30")
> grepl("^([+-]?)((\\d+)|(\\d*[.]\\d+))([eE]([+-])?\\d+)?$", test.values)
[1] TRUE TRUE TRUE TRUE

Update: suppressWarnings(), for instance. I did overlook that one. However, what you get there is suppression of all warnings, not only the one you wanted to suppress. Typically, the range of warnings is not defined by the documentation of the function guarded by this function... so you never know what you got rid of.

Seems like a built in function would be a beneficial addition.

Python console progressbar "???"

1%^M142/6456


Hence, 142 / 6456 < 0.02.

Good old Python (???) progressbar .. broke the mold, eh? Don't want to know ;)

No swap on AWS

Using Linux on AWS? As of now, there's no swap file then by default coming out of the images.

See the below two links for advices

How can I check if swap is active from the command line?
Why don't EC2 ubuntu images have swap?

A brief highlight roughly from those is this helpful command, to confirm your suspicion:

cat /proc/meminfo | grep -i swap

This will likely provide you with big fat zeroes as below:

SwapCached: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB

How cool is that (without a warning at the very least)...

Watch out when doing memory intensive processing (which is a less than good sign :) but happens), for any reason.