↓ Twitter is updated more often, so read it! ↓

Archive for the ‘programs’ Category.

Single-pass standard deviation in PHP

I recently had a need for a standard deviation function for a project. It wasn’t until after I’d already implemented it that I discovered the existence of a virtually-undocumented stats_standard_deviation() in PHP’s Statistics package.

Anyway, I did some research and found Don Knuth’s on-line, single-pass standard deviation function and implemented it in PHP.

<?php
function stddev($array){
  //Don Knuth is the $deity of algorithms
  //http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#III._On-line_algorithm
  $n = 0;
  $mean = 0;
  $M2 = 0;
  foreach($array as $x){
      $n++;
      $delta = $x - $mean;
      $mean = $mean + $delta/$n;
      $M2 = $M2 + $delta*($x - $mean);
  }
  $variance = $M2/($n - 1);
  return sqrt($variance);
}
?>

Please note that there are more efficient ways of doing this if you have have written a mean() or variance() function, as you should. There’s also a stats_variance() function in PHP.

I think I might do some serious benchmarking with PHP’s implementation of stddev and mine and see which is faster. I’ll be that the PHP one is faster because it’s written in C and mine’s being interpreted at runtime.

Update 19:40 stats is undocumented because it’s a PECL extension which isn’t included by default, and it’s not terribly clear to install. Install php-dev then use pecl install stats to get it. Don’t forget to sudo if you need to.

Update 2009/11/02 14:52 I had a missing -1 on the variance calculation for nearly a year and finally someone caught it. Thanks, Shane.

Government as open source software and development in general

Warning: this article gets a little ranty, but please, bear with me and help improve my thoughts by commenting.

Doc Searls of Linux Journal linked recently in his article Is government open source code we can patch? to an article by Britt Blasser entitled “Oh, if only government went in for an open source make-over…”. The article indirectly cites through a reference to Phil Hughes’ own Our Internet article two articles, FCC: Moving Beyond Network Neutrality and Our Internet!, by Bob Frankston, who Doc Searls recently interviewed for Linux Journal in Beyond Telecom (non-free registration required, subscribers can get it free).

In his article, Searls says:

Democracy is by nature “our government”. The open source twist on that we put it together and can hack improvements to it. Think of elected officials as committers and maintainers and you start go get the idea.

The analogy isn’t perfect, because by nature open source code is purely practical: it has to work. While government often does not. All government is buggy. In the worst cases it crashes outright and is replaced or supplemented by corrupt alternatives.

This analogy is fairly strong. However, commenter Frymaster supplies an addendum which strengthens it:

The US Constitution itself is open source, if you will, and editable. “The Framers” intended that Americans would change it to meet changing times, hence the series of amendments covering key rights like voting, and, most importantly, consuming alcohol. But they set the bar high, requiring super-majorities in both houses of Congress PLUS each of the states.

The Constitution certainly is open source. We’re free to change it, and other countries, fledgling or long-established, are welcome to take our code—our elemental specification of government—and adapt, implement, and utilize it. We’d like to receive contributions back, because they might be worthwhile enough to include in the trunk code. Even if these modifications aren’t strong enough for trunk, they might be strong enough for that government to maintain as a branch.

The Constitution is inherently good. Some might argue that it is outdated, but these folks are in error and their sentiments should be dismissed if they believe that it is irrelevant and should not be followed. The Constitution establishes a rule of law, wherein all citizens of the land give rights to a union of states, called the United States of America. It establishes procedures for updating it through amendments, which require a majority vote not only by two small bodies of people, but a majority vote of the several states, as Frymaster reminds us. This amendment process keeps those two smaller bodies of 535 people from legislating away the rights of their constituents.

I digress.

A government can never truly be open source, at least realistically. There will always be secrets; unpublished code, per say. These secrets are matters of national defense: military operations during a time of declared war, location of the president and vice president so the two are rarely together (for obvious reasons).

However, most of the goings-on of the government should be open. Obama pushes for “open government.” Ron Paul pushes for less government and, presumably, open government, since there would be far less government to hide!

However, open source projects thrive on the involvement of the people. Our current election system does not encourage responsible voting. Take, for example, Pennsylvania’s primary election results. 90,836 PA Republicans voted for Mike Huckabee, who dropped out of the race March 4, 49 days prior to the election. This is like allowing all of the Linux users in PA, regardless of technical knowledge, vote on the addition of one of three kernel features, and giving them only the name of the feature—no description, background, author, codebase, language, performance evaluation, or source! Even more appropriate would be that the developer of the feature conceded that one of the other two features is more efficient and worthwhile than his own!

I’m not saying that only those in-the-know should be allowed the vote. Mistaking my words for that would be fallacious. I believe that every person deserves the right to vote. I believe that every person has the right to have evidence of their vote. I believe that every person has the right to request vote totals for every level of complexity in the elections system: precinct, county, congressional district, state, and federal.

As versatile and open as the Constitution may be—and yes, I believe that it may need some updating to reinforce personal liberty and states’ rights—the government and governance which exists now cannot be patched.

When working on a project, a developer comes to a point where he or she realizes that there is a major defect in the software. He or she (for sake of my fingers, I’m going to use he henceforth, pardon my faux pas) has two options: patch or rewrite. He knows there are serious bugs, bugs which are inherent in the design of the code, as it has been patched since it was written. These most of these patches were good things, but some introduced more bugs which have yet to be fixed.

Should the developer continue to patch the code? Or should he rewrite it, integrating the features of the old version with more manageable code and lessons learned since it was first written?

If he continues to patch it, he treats the problem, but may not actually fix the problem. The problem might be inherent, or the problem may be caused by a combination of other features.

If he rewrites it, he’ll spend a lot of time redoing work he’s already done, but the result will be a more efficient program with fewer bugs (hopefully) and more manageable, cohesive code base. Of course, this new version will have its bugs, too, but they may be easier to find if the code is more manageable.

What the United States needs is a rewrite. The foundations of our government are solid: the Constitution, capitalism and free market economy, liberty for all. These are like the basic functions that make a program tick, “the algorithm,” per say. There are other parts that work, too. However, the maintainers of our government have diverted our attention from the core of the government, preferring us to view the only the parts it wants us to see, i.e. the bling.

The founders—the original authors of the U.S. base code—have long since gone, and they would be disgusted by the mess that is the program and its code base—the federal government—today. Things aren’t going they way that they would have wanted, and there’s a growing part of the population which seeks to return the these ideas, to a set of ideas not much different in theory from the software philosophy of “do one thing and do it well.” The government is bloated, more bloated than any application to which I might possibly compare it.

The founders outlined in the Constitution—a white paper or base algorithm, if you prefer to keep up the programming analogies—the specific functions of the U.S. government. It outlined a process by which the white paper or algorithm itself could be changed to allow more functions or remove functions which endangered the users and their data. However, rather than use this process, the maintainers simply implement functions or remove functions without regard to the users’ opinions, data, or the fact that the users are supporting the maintainers financially.

A fork is nearly impossible. There is no more undiscovered land on the face of the Earth, and it would be difficult for a state to secede. It would be economically infeasible, as well as diplomatically infeasible. The citizens of the new country would have their rights and their government as the Constitution of the U.S.—and of their new country—defines, but they would not have the resources to be self-sufficient, and things imported would cost more.

Fortunately, our government is not setup like the Linux kernel maintenance group in that The President of the United States is not a benevolent dictator (at least not on paper, haha). There are, however, 50 branches of the U.S. kernel which all have their own quirks and maintainers. Like a few projects, perhaps the Debian project included, the U.S. government does have representatives, people who supposedly represent their constituents in the decision-making process having been chosen by those constituents.

The ears of the maintainer are closed. He doesn’t want hear from the users; the people. However, the representatives’ ears are open, and they must be, by definition. In order to change the course of the program; the United States, we, as users; citizens, must elect representatives who believe as we do and not settle for anyone lesser. These representatives will then make the decisions we would make if we were in their position, leading to a program which is more useful, usable, smaller, and most of all, open.

HOWTO Update Twitter from the console or GUI

I’ve gone and added to the script, checking for errors and making it more slick.

Check it out at twitter-gx or read the code contained herein.


#!/bin/bash
#@Name: twitter-gx
#@Author: Colin Dean <cad@cad.cx>
#@URL: http://cad.cx/twitter
#CHANGE THESE
EMAIL=you@yours.com
PASSWORD=lolz
#PATHS
ZENITY=/usr/bin/zenity
CURL=/usr/bin/curl
ICON=/usr/share/pixmaps/gnome-irc.png
#can someone suggest a better icon?
#STRINGS--someone want to localize?
WHAT="What are you doing? "
SORRY="Sorry, that was too long."
UPDATED="Tweet submitted:"
FAILED="Tweet failed. Check output for why."

CURL_MISSING="curl was not found at ${CURL}. Please install curl."
ZENITY_MISSING="zenity was not found at ${ZENITY}. Please install zenity."
#-------Woe to those who make changes below unless they know what it does------#

#META
NAME="twitter-gx"
VERSION="1.0"
URL="http://cad.cx/twitter/twitter.xml"
#functions galore
function check_for_deps {

#I'm not going to bother checking for awk. Does any sane distro omit it?
  if [ ! -x ${CURL} ]; then

    CURL=`which curl`
    if [ "$?" = 1 ]; then

      warn_user "${CURL_MISSING}"
      exit 1
    fi
  fi

  if [ ! -x ${ZENITY} ]; then

    ZENITY=`which zenity`
    if [ "$?" = 1 ]; then

      warn_user "${ZENITY_MISSING}"
      exit 1
    fi
  fi

}

function getinput {
  if [ ${TERM} = "xterm" ]; then echo -n "${WHAT}"; read TWEET
    else TWEET=$(${ZENITY} --entry --text="${WHAT}" --entry-text="${1}")

  fi
}

function check_length {
  TWEETLENGTH=$(echo -n "${TWEET}" | wc -m)

  if [ ${TWEETLENGTH} -gt 140 ]; then

    warn_user "${SORRY} (${TWEETLENGTH} > 140)"
    getinput "${TWEET}"

    check_length
  fi
}

function warn_user {
  if [ ${TERM} = "xterm" ]; then echo "${1}"

    else ${ZENITY} --warning --text="${1}"
  fi

}

function urlencode () {
#one day, I will understand awk
#got this from somewhere
  echo "$@" | awk '

  BEGIN {
   # with inspiration from Heiner Steven and Rick Richardson
      split ("1 2 3 4 5 6 7 8 9 A B C D E F", hexit, " ")
      hexit [0] = 0 # setup the hex values we will use..
      for (i = 1 ; i <= 255 ; ++i) ord[ sprintf ("%c", i) "" ] = i + 0

    } {
      encoded = ""
      for (i = 1 ; i <= length ($0) ; ++i) {
        c = substr ($0, i, 1)
        if ( c ~ /[a-zA-Z0-9.-]/ ) { # allow only basic chrs
          encoded = encoded c

        } else { # encode everything else..
          lo = ord[c] % 16
          hi = int (ord[c] / 16);
          encoded = encoded "%" hexit [hi] hexit [lo]
        }

      }
#printf encoded
    }
  END {
    printf ("%s", encoded)
  }

'
}

function submit_tweet {
  OUTFILE="/tmp/twitter_response-`date +%Y%m%d%H%M%S`.txt"

  ${CURL} -s \
       -u ${EMAIL}:${PASSWORD} \
       -o ${OUTFILE} \
       -d status="$(urlencode "${TWEET}")" \
       -H "X-Twitter-Client: ${NAME}" \
       -H "X-Twitter-Client-Version: ${VERSION}" \
       -H "X-Twitter-Client-URL: ${URL}" \
       http://twitter.com/statuses/update.xml

}

function notify_user {
  if [ "${1}" = "success" ]; then

    TERM_MESSAGE="${UPDATED} ${TWEET}"
    GUI_MESSAGE="${UPDATED}\n\n${TWEET}"

    ICON_STR="--window-icon=${ICON}"
  fi
  if [ "${1}" = "failed" ]; then

    TERM_MESSAGE="${FAILED}"
    GUI_MESSAGE="${FAILED}"
    ICON_STR=""

  fi

  if [ ${TERM} = "xterm" ]; then echo "${TERM_MESSAGE}"

    else
      exec 3> >(${ZENITY} --notification --listen ${ICON_STR})

      echo "message: ${GUI_MESSAGE}" >&3
      sleep 10
      exec 3>&-
  fi

}

TWEET=$@ #see if it was supplied on the command line
if [ "${TWEET}" = "-v" ]; then echo "${NAME} ${VERSION}" && exit; fi

check_for_deps
if [ -z "${TWEET}" ]; then getinput; fi #if it wasn't, prompt for it

check_length #make sure it's less than 140 characters
submit_tweet #submit it since we got past the check

if [ "$?" == 0 ]; then notify_user success && exit 0;

else notify_user fail && exit 1; fi

HOWTO Write a script to update Twitter from Linux

I just signed up for Twitter as colindean. I don’t know what use I’ll have for it, as I’m not one who is much for microblogging or partying.

Anyway, the API caught my eye, and after having read Wayne’s post at Fsckin w/ Linux regarding Twitter clients for Linux, I decided to roll my own rather than use what’s out there. I wanted something simple and bashful.


#!/bin/bash
EMAIL=youremail@yourdomain.com
PASSWORD=yourpassword

WHAT="What are you doing? "
SORRY="Sorry, that was too long."
UPDATED="Tweet submitted:"

function getinput {
  if [ ${TERM} = "xterm" ]; then echo -n "${WHAT}"; read TWEET
    else TWEET=$(zenity --entry --text="${WHAT}" --entry-text="${1}")
  fi
}

function check_length {
  TWEETLENGTH=$(echo -n "${TWEET}" | wc -m)
  if [ ${TWEETLENGTH} -gt 140 ]; then
    warn_user
    getinput ${TWEET}
    check_length
  fi
}

function warn_user {
  if [ ${TERM} = "xterm" ]; then echo "${SORRY}"
    else zenity --warning --text="${SORRY}"
  fi
}
function submit_tweet {
  curl -u ${EMAIL}:${PASSWORD} \
       -d status="${TWEET}" \
       http://twitter.com/statuses/update.xml \
  > /dev/null
}

TWEET=$@ #see if it was supplied on the command line
if [ -z "${TWEET}" ]; then getinput; fi #if it wasn't, prompt for it
check_length #make sure it's less than 140 characters
submit_tweet #submit it since we got past the check

if [ ${TERM} = "xterm" ]; then echo "${UPDATED} ${TWEET}"
  else
    exec 3> >(zenity --notification --listen --window-icon=/usr/share/pixmaps/gnome-irc.png)
    echo "message: ${UPDATED}\n\n${TWEET}" >&3
    sleep 10
    exec 3>&-
fi

Just replace the EMAIL and PASSWORD variables and drop it into a directory in your $PATH. You can add it to the panel, too—it will detect if it’s been executed at a terminal or from the panel or other GUI element. It could also very easily be internationalized.

Let me know what you think of this script—it’s 4:00 am and I’m about to pass out.

Sorry, that one line might overflow the box on slim resolutions. I’ll get that fixed one of these days. Update: That was an easy fix. Added overflow: auto; to the codeblock CSS.

Creative crippled Vista X-Fi driver, orders modder who fixed it to cease and desist

Creative Labs message boards user Daniel_K had been modifying the Vista driver to re-enable functionality which was apparently available on Windows XP but disabled for Vista. In a somewhat DMCA-like claim this past Friday, Phil O’Shaughnessy, VP Corporate Communications at Creative Labs, wrote a message on Creative’s message boards asking using user Daniel_K “to respect our legal rights in this matter and cease all further unauthorized distribution of our technology and IP.” This created an immense outcry from folks who have paid a premium for Creative’s X-Fi line of sound cards, which is “intentionally crippled” on Vista. Creative doesn’t seem to care, though. “If we choose to develop and provide host-based processing features with certain sound cards and not others, that is a business decision that only we have the right to make.” It appears that the modified drivers are still available, but will no longer be updated as per Creative’s request.

I wish Creative would release a fully-working Linux driver. Even though I did get the Linux X-Fi driver working, Ubuntu still freezes 2 out of 5 times I boot and even then some sounds are completely distorted, such as Pidgin’s.

What is happening to the computer hardware industry? Seriously. nVidia’s crappy drivers drove me bonkers, and from Windows 32-bit to 64-bit, where I’ve had almost no problems. Creative’s shenanigans will keep me off of Vista, and even the modded drivers won’t work because I have 8 GB of RAM, and there’s a note in the driver package which says that the drivers will crash the computer if there is more than 2 GB of RAM available.

I’m surprised that Creative doesn’t have a class-action lawsuit against it for false advertising. It touts features of the card on the box, but these features aren’t available on all of its supported platforms, and there’s no note saying that. Tsk tsk, Creative. You get a wag of the finger today.

HOWTO: Astronomy Picture of the Day for GNOME

Astronomy Picture of the Day is a really neat page which is generated daily by NASA. It shows a single picture captured by some NASA-affiliated person and some text to describe it.

These pictures are perfect for desktop backgrounds. There’s a program to automatically set it as the background for Windows, but we Linux, or more specifically, GNOME, users can do it in a simple script.

Drop this script somewhere on your system and create a cron schedule with crontab -e for it. The cron lines are in the script. I’d recommend the four hour intervals if you don’t keep your computer on all the time, or the twelve or once-a-day intervals if you do.

Oh, and you might need to install cron. Gutsy doesn’t have it by default, so you can do it through Synaptic or using sudo apt-get install cron at a command line.


#!/bin/bash
#Filename: apodwallpaper
#Location: ${HOME}/.bin
#Purposes: Downloads NASA Astronomy Picture of the Day and displays it as the
#          GNOME background 
#Author(s): acvwJosh of Ubuntu Forums, modified by Colin Dean <http://cad.cx>

#add the following line to your crontab if you want to run this hourly
#10   *  *   *   *     apodwallpaper

#add this line if you want to run it every four hours
#10 0,4,8,12,16,20 * * * apodwallpaper

#add this line if you want to run it every twelve hours
#10 0,12 * * * apodwallpaper

#add this line if you want to run it daily 
#10 0 * * * apodwallpaper

#don't forget to put this script in your $PATH.
#I usually add ${HOME}/.bin to my path in .bash_profile and put my scripts there

#change this if you want the image to be in a different directory
FILENAME=apodwallpaper
APODWALLPAPER=${HOME}/.${FILENAME}

mkdir -p ${APODWALLPAPER} && cd ${APODWALLPAPER}

# download image from apod site
wget -A.jpg -R.txt -r -l1 --no-parent -nH http://antwrp.gsfc.nasa.gov/apod/astropix.html

# move image from obscure folder to main folder, rename image
find ./apod -name "*.jpg" | while read line ; do
mv "$line" "${FILENAME}.jpg"
done

# set image to wallpaper
gconftool-2 -t string -s /desktop/gnome/background/picture_filename "blank.jpg"
gconftool-2 -t string -s /desktop/gnome/background/picture_filename \
                         "${APODWALLPAPER}/${FILENAME}.jpg"
gconftool-2 -t string -s /desktop/gnome/background/picture_options "zoom"

#get rid of cruft
rm -rf apod robots.txt

There’s a thread about this functionality at Ubuntu Forums entitled, APOD? anyone.

HOWTO: Creative X-Fi sound card on Ubuntu Gutsy

I know Hardy is due out in less than a month’s time, but I got a Creative Sound Blaster X-fi XtremeGamer on super rebate a few weeks ago and, dammit, I want sound on Linux!

Unfortunately, Creative only recently released specifications to OSS and ALSA. OSS has a driver already in its newest release, but OSS and Ubuntu have never played well together. Most of Ubuntu’s applications prefer ALSA or ESD. Creative did release an ALSA driver a while ago, but only for 64-bit distros.

Fortunately, I’m running Ubuntu Gutsy amd64, so I can use the driver. Fear not, though, one user on Ubuntu Forums has tried it on 32-bit and reported success. He’s refining the process, though, and figuring out how exactly he did it.

NullHead, wolfc, and some others on Ubuntu Forums refined a process by which folks can install the Creative X-fi driver. Unfortunately for Gutsy-using folks, the process requires a kernel recompile because the Gutsy kernel uses SLUB instead of SLAB.

However, the process does work. After compiling the kernel, there’s a a few simple steps and a single patch which customizes the source for Ubuntu users.

Read the thread at Ubuntu Forums entitled [HOW-TO] X-fi beta driver.

It’s really nice to have sound in Ubuntu after not having it for approximately two weeks. I’ve been hopping onto another computer when I need it and using that other computer as my jukebox.

PulseAudio is the new default sound server in Ubuntu Hardy (8.04 for those readers who are out-of-the-loop), but it retains its compatibility with ALSA, OSS, ESD, and various other sound servers in Linux. Hopefully, ALSA will have a driver out soon.

Apparently, a company called Auzentech is working on higher-quality drivers for Windows for its licensing of Creative’s X-Fi technology. It’s also working on Linux drivers, but the ETA isn’t until Q4 2008. If Auzentech can pull it off, you bet your sweet bippie I’ll trade my XtremeGamer for one of its cards. I like to support those who support Linux :-)

HOWTO: Fixing gnome-vfs ssh support in Firefox

In theory, the package firefox-gnome-support handles GNOME’s Virtual File System URLs. However, out-of-the-box, it lacks support for the ssh handler used when connecting to something via ssh.

To fix this, go to about:config and find the key called network.gnomevfs.supported-protocols. Append ;ssh:, thus changing it to smb:,sftp:,ssh:. It will magically work!

Why forum lockouts are bad for business

I believe that most bulletin board software packages offer a lockout feature. A user can be locked out when attempting to log in if they—or a malicious person—input an incorrect password so many times.

This is well-and-good in the cases when Mallory is trying to get into Alice‘s account.

However, it is very, very annoying for a legitimate user who has forgotten his or her password. By the time he or she waits the 15 minute lockout period, he or she may have forgotten the comment or question he or she wanted to post. The user loses out on an answer, the community loses out on a comment, and the site loses out on increased dialog between users—the main thing that drives a site.

When a user resets his or her password using the “forgot my password” feature, that lockout should reset. Most of the time, it doesn’t, and that’s bad for business. It’s punishing the user, and a program should never punish the user.

HOWTO: Reset a forgotten MySQL root password

Every system administrator has done it at some point: forgotten the root password for MySQL. It’s aggravating because it was previously difficult to reset. However, it is quite easy now.

Execute this a command line to reset the root password.

sudo /etc/init.d/mysql stop
sudo mysqld_safe --skip-grant-tables
mysql --user=root mysql

Once you’re in the mysql shell, execute these queries.

update user set Password=PASSWORD('the-new-password');
flush privileges;
exit;

Thanks to Howtoforge for pointing this out.