↓ Twitter is updated more often, so read it! ↓

squid-deb-proxy, or how Ubuntu updates fly at work

I recently got permission at work to install squid-deb-proxy in order to speed up updates for Ubuntu users and reduce the amount of bandwidth we consume every morning when Update Manager checks for updates. There are 12 of us using it now, and the number generally grows every second or third hire.

squid-deb-proxy is a specific configuration of squid, an HTTP proxy, which enables caching of retrieved files and limits access to only the official Ubuntu mirrors. The net result is that client machines look at the server running squid-deb-proxy for updates, and if the proxy has recently downloaded the file, it’s sent to the user immediately, or, if it’s not in the cache, the server retrieves and caches it, then passes it to the client.

The business benefit is that it saves considerable amounts of time because it saves bandwidth, especially around upgrade time. Imagine 12 people pulling down ~50 MB of updates on a given day. Now, imagine 12 people upgrading to the latest Ubuntu release come April or October–now we’re talking one or two gigabytes of updates. If those updates are available locally, the bottleneck moves from the Internet connection to the sheer speed of the computer. The user is back up and running sooner, and thus restored to productivity sooner.

Installing the server component of squid-deb-proxy is no more difficult than sudo apt-get install squid-deb-proxy. It’s up and ready to go upon installation.

There are two ways for the clients to get updates. One is by manually specifying the proxy in the apt configuration by adding this block to /etc/apt/apt.conf.d/99proxy:

Acquire {
 Retries "0";
  HTTP { Proxy "http://YOUR_PROXY_SERVER_HOSTNAME_OR_IP_HERE:8000"; };
   };

The other way is much cleaner, but the server must be on the same subnet as the clients. Simply install squid-deb-proxy-client with sudo apt-get install squid-deb-proxy-client. Warning: if you have both set up, squid-deb-proxy-client will prevail.

Courtesy of two changes I proposed and which were recently merged into the Natty cycle (Ubuntu 11.04), the configuration options to allow but not cache unspecified domains and add extra mirrors are present, increasing the out-of-the-box usefulness and easy configuration of the server component.

So, squid-deb-proxy is worth it for us. It saves time and alleviates the morning bandwidth burst.

We used it at my previous job, too, and it was worth it with only four Ubuntu users hitting it.

If you use it, how many people do you have hitting it?

6 Comments

  1. Jonathon:

    YES!!!! Creating the 99proxy file like you said worked!!!!! I have been trying to fix this f-o-r-e-v-e-r!!! THANK YOU SO MUCH!!

    =D

  2. Colin Dean:

    I’m glad you found the article useful, Jonathon.

  3. ehcache.net:

    squid-deb-proxy, or how Ubuntu updates fly at work…

    I recently got permission at work to install squid-deb-proxy in order to speed up updates for Ubuntu users and reduce the amount of bandwidth we consume every morning when Update Manager checks for updates. There are 12 of us using it now, and the numb…

  4. Stefan Benediktsson:

    We have a couple of thousands Ubuntu clients at my company (managed by cfengine3) and I’ve been playing around with services like apt-cacher-ng and apt-proxy, but none of them scaled very well. Previously we did a full debmirror (Ubuntu 8.10, 9.10, 10.04 for both x86 and amd64). This generated about 170GB download/day.

    In my new setup I have an Apache configured as a reverse proxy where I map all the external/internal repositories I want to use. The Apache server will then forward the requests to squid-deb-proxy which populates my local mirror.

    client => [ apache => squid-deb-proxy ] => outgoing proxy => INTERNET => repository

    By doing this I have now reduced the external download bandwidth to about 640MB/day with a total disk usage of 7.5GB and a cache hit rate of 99.9% !!!

  5. Stefan Benediktsson:

    Updated status: We are now running about 4500 active clients world-wide against a backend cache structure of three (!) apache/squid-deb-mirror servers.

    CPU load: ~4.0% / Mem cache: 8GB / Disk cache: 5.2GB

  6. Gaurav Sharma:

    Hi,

    I recently used this method for updates in my company. It worked well on the company network, however when I use my system at home the apt-get doesn’t work at all.

    I am using the 99proxy method as the server is located on a different subnet in the company’s network. Can you suggest something that can make it work at home internet connection as well?

    For your reference, here is the output of apt-get update command:

    gaurav@Gaurav-Sharma:~$ sudo apt-get update;
    [sudo] password for gaurav:
    0% [Working]Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover
    Failed to exec method Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover/usr/share/squid-deb-proxy-client/apt-avahi-discover

    Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover
    Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover
    Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover
    Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover
    Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover
    Failed to exec method /usr/share/squid-deb-proxy-client/apt-avahi-discover
    0% [Connecting to 192.168.1.250 (192.168.1.250)] [Connecting to 192.168.1.250 (192.168.1.250)] [Connecting to 192.168.1.250 (192.168.1.250)] [Connecting to 192.168.1.250 (192.168.1.250)]

    Here 192.168.1.250 is proxy server’s IP.

    Thank you.

Leave a comment