The Flow of Consciousness

I recently got permission at work to install squid-deb-proxy in order to speed up updates for Ubuntu users and reduce the amount of bandwidth we consume every morning when Update Manager checks for updates. There are 12 of us using it now, and the number generally grows every second or third hire.

squid-deb-proxy is a specific configuration of squid, an HTTP proxy, which enables caching of retrieved files and limits access to only the official Ubuntu mirrors. The net result is that client machines look at the server running squid-deb-proxy for updates, and if the proxy has recently downloaded the file, it’s sent to the user immediately, or, if it’s not in the cache, the server retrieves and caches it, then passes it to the client.

The business benefit is that it saves considerable amounts of time because it saves bandwidth, especially around upgrade time. Imagine 12 people pulling down ~50 MB of updates on a given day. Now, imagine 12 people upgrading to the latest Ubuntu release come April or October–now we’re talking one or two gigabytes of updates. If those updates are available locally, the bottleneck moves from the Internet connection to the sheer speed of the computer. The user is back up and running sooner, and thus restored to productivity sooner.

Installing the server component of squid-deb-proxy is no more difficult than sudo apt-get install squid-deb-proxy. It’s up and ready to go upon installation.

There are two ways for the clients to get updates. One is by manually specifying the proxy in the apt configuration by adding this block to /etc/apt/apt.conf.d/99proxy:

Acquire {
 Retries "0";
  HTTP { Proxy "http://YOUR_PROXY_SERVER_HOSTNAME_OR_IP_HERE:8000"; };
   };

The other way is much cleaner, but the server must be on the same subnet as the clients. Simply install squid-deb-proxy-client with sudo apt-get install squid-deb-proxy-client. Warning: if you have both set up, squid-deb-proxy-client will prevail.

Courtesy of two changes I proposed and which were recently merged into the Natty cycle (Ubuntu 11.04), the configuration options to allow but not cache unspecified domains and add extra mirrors are present, increasing the out-of-the-box usefulness and easy configuration of the server component.

So, squid-deb-proxy is worth it for us. It saves time and alleviates the morning bandwidth burst.

We used it at my previous job, too, and it was worth it with only four Ubuntu users hitting it.

If you use it, how many people do you have hitting it?

A Facebook friend of mine posted a story in the Telegraph about the British government wanting “to track every phone call, email, text message and website visit made by the public if they argue it is needed to tackle crime or terrorism”.

Conspiracy theories and tinfoil-hat-ism aside (“They’re already doing it! Every government! Even the US!”), this lead me to ponder a bit and consider a new market:

Encrypted personal email servers.

I wonder what people would pay for a service like that. Each person gets their own “server”, likely a VPS. Mail inbound or outbound wouldn’t be encrypted at the transport or application level, but perhaps mail to other servers in this network would be encrypted in transit.

Mail stored on disk would be encrypted, but the user would have to input a password in order for the storage area to be accessible whenever the server started up. A caveat would be that the encrypted storage area is accessible while the “server” is powered on–however, the moment it is shutdown, the data would be encrypted and require the user’s password.

On top of all of this, we urge users to utilize PGP encryption on all of their email. PGP keeps the messages themselves secure, and the security on the server keeps them doubly secure.

$35/mo per server up to perhaps 10 addresses? Also use it as a certain amount of file storage through WebDAV or the like?

Help me think out the logistics, folks who are more experienced than I.

Meta note: a series of full posts coming in the next week or two regarding travel–there’s a reason I’ve been silent!

VLC 1.1.0 was just released few hours ago. VLC is a fantastic media player and I use it on all of the operating systems I use (Ubuntu Linux, Mac OS X, Windows 7). It’ll play just about any format and do it wonderfully. It’s easier to use than Rhythmbox, Quicktime, or Windows Media Player for watching movies or iTunes for quick listening of an audio file.

Anyway, among new features are VP8 support (Google’s newly open sourced video format), lossless MPEG-4 support, AMR-NB support for audio in cellphone video, CDDB/CDTEXT on Windows when listening to CDs, and lots more.

A big feature in this release of VLC is the ability to decode video using the computer’s graphics card. You’ll need an nVidia graphics card to really benefit from it, though.

I decided to give it a quick test myself before upgrading from VLC 1.0.5 on my Windows 7 box. I played back a 720p HD rip of Adventureland and Back to the Future 3. Both were encoded with 23.97 fps H264 for video. Adventureland used DTSfor audio, while Back to the Future 3 used a52. I watched about 5 minutes of each and watched the Task Manager. For both movies, the CPU usage stayed between 15% and 30%, while memory usage stayed around 92,000 kilobytes.

Then, I upgraded to VLC 1.1.0 and enabled GPU support in the preferences. I watched the same parts. Both videos used the slightly less CPU, but less memory.

“What gives?” I thought. So, I did some research. I found the wiki article on VLC GPU Decoding and found that a video card using VP2 or newer is required. There’s a link to a table on Wikipedia showing the version of PureVideo support in each video card. Unfortunately, my 8800 GTX doesn’t support VP2.

Oh well, I guess I get to wait for the next video card I buy. However, if you own a video card made in the last 3 years, you probably have support for it, so go forth and enjoy low CPU usage when you’re watching videos.

I recently had a need to decompose a URL, do some stuff with the parameters, and reassemble it. I later had a need to take another URL and create a form on the fly, because using GET could overrun the URL character limit imposed by Internet Explorer.

Here’s my code for a decomposition and composition pair, plus the code to create a form dynamically using the output of the url_decompose() function.

/**
 * @param String url the URL to be decomposed
 * @return Object with full path in .path, params in arrays in .params
**/
function url_decompose(url){
    if(url.indexOf('?') == -1){ return {path: url, params: {} }; }
    path = url.substring(0, url.indexOf('?'));//get the path before the ?
    qs = url.substring(url.indexOf('?')+1);
    qsa = qs.split('&');
    params = new Object();
    for (var kv in qsa) {
        var kva = qsa[kv].split('=');
        var k = kva[0];
        var v = kva.slice(1, kva.length).join("=");
        params[k] = params[k] || new Array();
        params[k].push(v);
    }
    return {path: path, params: params};
}
/**
 * @param Object output of url_decompose()
 * @return String reassembled URL
**/
function url_compose(urlObject){
    if(!urlObject.path){return urlObject;}
    var urlString = '';
    var params = [];
    for (var kv in urlObject.params){
        if(kv.length > 0){
            params.push(
                encodeURIComponent(kv) +
                '=' +
                encodeURIComponent(urlObject.params[kv][0]));
        }
    }
    urlString = urlObject.path +
                    (params.length > 0 ? '?' + params.join('&') : '');
    return urlString;
}
/**
 * @param String path the path to be accessed
 * @param Array params associative array or object w/ params in key/value pairs
 * @param String method null for post or whatever method the form should use
**/
function do_post(path, params, method) {
    method = method || "post";
    var form = document.createElement("form");
    form.setAttribute("method", method);
    form.setAttribute("action", path);

    for(var key in params) {
      for(var idx in params[key]){
        var hiddenField = document.createElement("input");
        hiddenField.setAttribute("type", "hidden");
        hiddenField.setAttribute("name", key);
        hiddenField.setAttribute("value", params[key][idx]);
        form.appendChild(hiddenField);
      }
    }
    document.body.appendChild(form);
    form.submit();
}

If I’ve omitted an edge case, please do let me know and I’ll amend my code. I really do need to get one of those code highlighter plugins installed now that I’m doing more code examples.

When using select elements on a page (also known as drop downs) as a navigation item, it’s possible that the selected option will persist if the user use’s the browser’s back button.

Consider this workflow:

1. User clicks on dropdown and selects an item.
2. Javascript executes, doing something like “window.location = document.forms[0].selectElement.value”
3. Browser goes to next page.
4. User clicks back button, browser goes to previous page.
5. User is confused because the dropdown item contains what they just selected, not what was there when the page loaded.

This is an unfortunate bug/feature in Firefox and Chrome. It actually behaves as one would think in IE and Safari.

The reason is because of Gecko and Webkit’s bfcache, a feature which caches forward and backward pages so that they do not have to be reevaluated in order to be rerendered to the screen. IE reevaluates to rerender. I’m not sure why Safari behaves.

The trick is to use window.onpageshow and window.onload to handle the value of the sortby.

You’ll want something like this:

window.onload = “restoreSelect();”;
window.onpageshow = “restoreSelect();”;

or whatever the equivalent is in your favorite Javascript framework.

Don’t store the value in cookies. It’ll create problems if a user is using multiple tabs. The value I needed is always in the URL, so I could use that for this particular instance.

This trick works for me in Firefox, Chrome, Safari, and IE. I have not tried it on Opera.

I ran into an accessibility issue today wherein a user tabbing through an interface would not be able to toggle collapsable/expandable divs. The problem was that the span on which the user would click was inaccessible via the tab key and would not respond when hitting enter. So, I added this little gem to fix the problem.

<xsl:attribute name="onkeypress">
  <xsl:text>var wx; if(window.event){ wx = event.keyCode; } else { wx = event.which; };</xsl:text>
  <xsl:text>if(wx){ this.onclick(); }; return true;</xsl:text>
</xsl:attribute>
<xsl:attribute name="tabindex">
  <xsl:text>0</xsl:text>
</xsl:attribute>

If you don’t understand XSL, here’s a translation. I added a tabindex attribute with a value of zero and an onkeypress handler which simply activates the onclick of the same element.

I really admire Facebook’s Friend Suggestions feature. It has on rare occasion shown someone to whom I’d really like to connect. Most of the time, though, it shows people who I don’t know at all and have maybe one mutual friend. So, it’s largely useless to me.

However, being the often mindless user, when I see a suggestion that may be relevant to my interests, I do visit the full friend suggestion page and see if there are any others. Rarely am I presented with anyone to whom I find a dreadful desire to connect. I still feel obligated to click that damned little [X] as if to say, “I don’t know this person.” Unfortunately, I have to repeat this approximately 25 times in order to full expunge this list of people.

So, I decided to write a little script which takes care of clicking that button for me. Navigate to the Find Friends page on Facebook and execute this script in your Firebug console. Or, prefix it with javascript: and execute it in the URL bar.

d = document.getElementsByClassName("fg_action_hide"); for (var i in d){ d[i].onclick() }

The first part gets an array (a list) of all of the clickable elements with that class, the class used to denote the [X]. The next part loops over them and executes each’s onclick() function just like as if you’d clicked it yourself.

Please note that I tested this ONLY on Firefox and Chromium, but it should work on any modern browser (read: anything but IE).

I’ve done a ton of programming this week outside of work. I feel like I’ve had at least five browser tabs open for Launchpad at all times! If you’re not familiar with the site, it’s a code hosting service for open source projects, most notably Ubuntu Linux.

I’ve done some Esperanto translations on Launchpad prior to this week. I translated a large portion of Akregator (RSS/Atom feed aggregation for KDE) and a few others. I’ve also filed and added to a bunch of bugs, many of which were cleaned up by the recent One Hundred Paper Cuts bugfixing effort by the Ubuntu development team.

This week, I was abnormally active. I had some downtime here and there at work, plus some open nights, so I took advantage of this time to make significant contributions to three projects: Gwibber, Astrid, and Celtx.

Gwibber

Gwibber is a microblogging client for microblogging services such as Twitter, Identi.ca, and Jaiku. It also supports Flickr, BrightKite, FriendFeed, Facebook, and more, including generic RSS and Atom feeds. It’s written in Python by Ryan Paul (@segphault) of Ars Technica fame.

I had two and a half contributions this week. One is a patch addressing bug #364303, to add linkification of stock ticker symbols, denoted by a dollar sign followed by the ticker symbol, e.g. $GOOG. Clicking on a linked symbol will launch the browser and point it to Google Finance. Originally, it linked to StockTwits, but that site refuses to support OTC stocks.

The second contribution is a complete Esperanto translation of the program. I have mixed feelings on translating programs since most Esperantists would be using their computer in their native language. I decided to do it because, well, Gwibber has just a few more than 115 strings, so why not? It took me approximately an hour and a half, and it was well spent.

That half contribution is a committed, but as-of-yet unproposed/merged patch to fix bug #364322, wherein Gwibber doesn’t load Identi.ca/Laconi.ca groups at all. The problem is that the constructors for identica.SearchResult and laconica.SearchResult expect a TwitterAPI-compatible dictionary, but are passed a standard RSS feed dictionary instead. identica.Client.get_group() hits an RSS feed instead of an API function, because the Groups API in Laconica is not yet finalized. This causes Gwibber to throw an Exception–it’s caught and displayed, but the results aren’t loaded. I modified identica.Client.group(), which calls get_group(), to instead call identica.Client.get_search() and load the search results for the group name. It’s not a clean result set, though, because Laconia while searching strips the “!” used to denote the group name.

Hopefully the API will make it into Laconica 0.8.1 (the next release, mid-to-late September if they release on their typical schedule), I’ll modify the functionality to use the API instead and it’ll be much cleaner and yield more accurate results.

Astrid

Astrid is a TODO list manager for Google Android. It synchronizes with the excellent Remember The Milk TODO list service and provides task reminders and such. I use it on my T-Mobile G1 and would certainly miss it if it were suddenly gone.

My contribution to Astrid was another (nearly) complete Esperanto translation. I left about 10% untranslated because, well, I needed to go to bed! Unfortunately, I realized approximately 3/4 through my translation that Astrid is unlikely ever to be on a platform which supports Esperanto. The chances of there ever being an Esperanto translation of Android are atomically small.

However, my work was not wasted. Launchpad’s Rosetta translation system permits projects to share their translation with other Launchpad projects, using a essentially a central database of translations. This enables projects to speed up internationalization by using existing translations which exactly match internationalize-able strings. While it’s unlikely that anyone will ever use Astrid in Esperanto, it is likely that another TODO list project hosted on Launchpad could benefit from my work in the future.

Celtx

Celtx is an integrated media pre-production and screenwriting utility. I’ve used it for writing short plays for classes and such. I’m using it now for a forthcoming project which I’ll eventually let out of the bag.

My contribution for Celtx, as the link above shows, has not been code or a translation, but a package for Ubuntu. This was the first time I’d ever build a Debian package (Debian packages are what Ubuntu, a Debian Linux derivative, uses to distribute programs).

A long-existing bug, #99965, requested that someone package Celtx from the source on its main web site. I took interest in it in November 2007, but my initial attempts to package it for Hardy (Ubuntu 8.04, released April 2008) were unsuccessful due to my inexperience with build systems and a lack of time/concentration. I revisited the task for Intrepid (Ubuntu 8.10, October 2008) and Jaunty (9.04, April 2009), but was too busy with my new job and other life happenings.

I painstakingly built Celtx from source manually several times before building a binary package myself without any help from the Debian packaging system’s helper scripts (debhelper, dh_make). It’s a pain in the ass because Celtx is based on Mozilla, the build process of which is not as straightforward as a ./configure && make && make install. However, once I had a working knowledge of the build process, I delved into debhelper and its kin and built a source package. I did it for Jaunty first, as that’s what version I’m running right now, but then I made some modifications here and there to convince Celtx to build on Karmic (9.10, due October 2009).

I’ve learned a ton about deb packages and about make, and I think I can consider myself a “packager” now, having done so.

I’m hoping that Celtx Studios, the company that produces Celtx, releases the source for version 2.0.2 soon. The 2.0.1 source is available, but out-of-date. I’d like to get 2.0.2 into Karmic, but it’s looking like 2.0.1 will have to do for now.

So, if you have an interest in media production and/or screenwriting, or wish to help test a new package for inclusion in the next version of Ubuntu, please, by all means, install my Celtx PPA for Ubuntu Jaunty or Karmic and test the crap out of the package. There are a few known issues on Karmic, but I’m not a C++ coder, so I don’t know what I’d even try to fix.

Moving Forward

I look forward to participating more in the development of Gwibber. Because of my fiddlings with it, I’ve come to know the DVCS tool Bazaar (bzr) pretty well, as Launchpad and it are heavily integrated (hint: Canonical, developer of Ubuntu, also developed Launchpad and Bazaar!).

I know that I’ll keep translating. I’ve noticed that my vocabulary has grown a bit and I’m not using the vortaro (dictionary) as much. I still reference Traduku.net, Lernu!, and Komputeko frequently to get technology terminology and infrequently-seen words.

I recently noticed that WordPress added support for Gears, as have a few other sites, including ZohoOffice and MySpace.

Gears is essentially offline storage for rich Internet application data. GoogleOS has a list of Gears-enabled sites/applications. Read more on Gears at Wikipedia.

Much to my dismay, the only officially-available Firefox extension supports only 32-bit operating systems. I’m a 64-bit Linux user. Specifically, I use Ubuntu 8.10 Intrepid with my ASUS M3A32-MVP motherboard and 8 GB of DDR2 RAM (4GB of Geil and 4 GB of OCZ).

Fortunately, Gears is an open source project and one intrepid developer made a Gears Firefox extension package which supports 64-bit Linux.

I downloaded and installed the package by dragging and dropping it onto the Firefox Addons dialog, then restarting Firefox when prompted. I archived the Gears extension package on my own site in the event the package is suddenly no longer available. Be sure to check for new versions though, since this package may be out of date if you’re reading this post more than a few days after publication (though unlikely).

Huzzah for open source.

Yesterday, I showed my implentation of Don Knuth’s on-line standard deviation implementation.

Today, I have for you, my interested reader, the results of a quick benchmark on my code versus the PECL stats package.

Here’s the script for the benchmark:

<?php
function stddev($array){
  //Don Knuth is the $deity of algorithms
  //http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#III._On-line_algorithm
  $n = 0;
  $mean = 0;
  $M2 = 0;
  foreach($array as $x){
      $n++;
      $delta = $x - $mean;
      $mean = $mean + $delta/$n;
      $M2 = $M2 + $delta*($x - $mean);
  }
  $variance = $M2/$n;
  return sqrt($variance);
}

function mt($precision=4){ return round(microtime(true),$precision); }
function echodiff($start,$end){ echo ($end - $start) . " μs to execute\n"; }

//first, let's prepare a set of arrays.
mt_srand();
//four arrays on different orders of magnitude
$array10 = array();
$array100 = array();
$array1000 = array();
$array10000 = array();
$array100000 = array();
$array1000000 = array();

for($i=0;$i<10;$i++) $array10[] = mt_rand(0, 100);
for($i=0;$i<100;$i++) $array100[] = mt_rand(0, 100);
for($i=0;$i<1000;$i++) $array1000[] = mt_rand(0, 100);
for($i=0;$i<10000;$i++) $array10000[] = mt_rand(0, 100);
for($i=0;$i<100000;$i++) $array100000[] = mt_rand(0, 100);
for($i=0;$i<1000000;$i++) $array1000000[] = mt_rand(0, 100);

$ars = array($array10, $array100, $array1000, $array10000, $array100000, $array1000000);
foreach($ars as $ar){
        echo "Trial of " . count($ar) . " elements.\n";
        $mine_start = mt();
        $mine_out = stddev($ar);
        $mine_end = mt();
        echo "Mine: ";
        echodiff($mine_start, $mine_end);
        $php_start = mt();
        $php_out = stats_standard_deviation($ar);
        $php_end = mt();
        echo "PHP/PECL: ";
        echodiff($php_start, $php_end);
}
?>

These are the results on my machine, an Athlon X2 6000+ with the max memory size in php.ini set to 2048M.

[colin@rahab Source]$ php stddev.php
Trial of 10 elements.
Mine: 9.98973846436E-5 μs to execute
PHP/PECL: 0 μs to execute
Trial of 100 elements.
Mine: 0.000200033187866 μs to execute
PHP/PECL: 0 μs to execute
Trial of 1000 elements.
Mine: 0.0019998550415 μs to execute
PHP/PECL: 0.000200033187866 μs to execute
Trial of 10000 elements.
Mine: 0.0218000411987 μs to execute
PHP/PECL: 0.00450015068054 μs to execute
Trial of 100000 elements.
Mine: 0.216400146484 μs to execute
PHP/PECL: 0.042799949646 μs to execute
Trial of 1000000 elements.
Mine: 1.13669991493 μs to execute
PHP/PECL: 0.213500022888 μs to execute

Approximately .8 μs isn’t much, but it could be higher if the dataset it much higher and the calculation is repeated several hundred times. Regardless, it shows that you should probably take the time to simply install PECL stats and use it instead of trying to roll your own.