Index¶

Making vim look like textmate (again!)

A few years ago I posted about my quest to make editing ruby in my terminal as beautiful as in Textmate. I've been back on linux for a year now but it's only tonight I've gotten around to sorting out my vim display.

RedCar, an open source cross platform alternative to TextMate, reminded me how pretty code can look. Tonight I worked out how to make vim on ubuntu lucid look just as lovely.

1. Grab the twilight256 colorscheme and drop it into ~/.vim/colors/twilight256.vim
2. Make sure you're using 256 color terminal (I put this in ~/.bashrc) export TERM=xterm-256color
3. Tell vim to use this colorscheme (I put it in ~/.vimrc) colorscheme twilight256
4. In my terminal under "Edit" -> "Profile Preferences" I selected "Deja Vu Sans Mono Book" "13"

I can't believe it took me so long to do this but I'm glad I finally have.

Ruby can look pretty under Vim on Linux

The Long Tail

One of the questions in a technical interview I sat today was "Write 'tail' in Ruby". I had a pen and paper and no access to Google. I think I came up with a reasonable solution but I realised how dependent I am on docs! It took me a long time to arrive at a solution when I couldn't refer to Ruby API docs.

#!/usr/bin/env ruby

filename = ARGV[0]

end_marker = 0
offset_increment = 1000

lines = []
fh = File.open filename
while lines.size < 10
  current_offset = end_marker + offset_increment
  fh.seek(-current_offset, IO::SEEK_END) || fh.rewind
  buf = fh.read(current_offset - end_marker).split("\n")
  lines.insert(0, *buf)
  end_marker += current_offset
end
puts lines.last 10

Hmm, it's still kinda broken.

Strange HTTP Performance Variations Between OS’s

[2013-12-19 I let latentsee.com lapse but you can still install latentsee on your own webserver]

In a recent talk at VelocityConf, John Rauser explained the effect of TCP Slow Start and Congestion Control on web performance. He pointed out that RFC 1122 states:

Recent work by Jacobson on Internet congestion and TCP retransmission stability has produced a transmission algorithm combining "slow start" with "congestion avoidance". A TCP MUST implement this algorithm.

While examining the impact of these with my new HTTP performance testing tool (LatentSee) I noticed that the charts generated on my Mac & Windows machines didn't seem to match the theory. Usually we would expect to receive 3 packets (< 4500 bytes) in the first segment. Instead I am seeing up to 67KB on the Mac and around 35KB on Windows 7.

Effect of file size on retrieval time from slicehost.latentsee.com

Original images: | Theory | Ubuntu | Windows 7 | Mac OS X

When I ran the same tests against a different hosting provider (Brightbox), Windows and Mac OSX behaved more like my Ubuntu box.

Effect of file size on retrieval time from brightbox.latentsee.com

Original images | Theory | Ubuntu | Windows 7 | Mac OS X

I'm very curious about these differences. It takes the same time for my Mac to retrieve any file up to 67KB in size from Slicehost. Have they tuned their TCP stack differently? Why then does Ubuntu behave similarly against both Slicehost and Brightbox? Is everyone conforming to the RFCs?

I'd love to hear if people can reproduce these results.

You can experiment with LatentSee using your web browser. - USA: slicehost.latentsee.com - UK: brightbox.latentsee.com - AUS: ultraserve.latentsee.com

You can also put latentsee.php on your Apache webserver to test your own servers (be sure to disable compression on the vhost).

TCP and the Lower Bound of Web Performance - John Rauser

I was fortunate enough to be in the room for this talk by John Rauser at Velocity 2010. I realized a few minutes in that this talk was (a) super interesting and (b) not being filmed. So I pulled out my Flip and caught most of it.

This talk inspired me to plot some real world data to test the theory.

The slides from the talk are here.

See Also: John Rauser's "Look At Your Data" from Velocity 2011

Latency is a Killer

If you're hosting your website overseas you might want to rethink. I've been doing some research on the true impact of hosting Australian websites in the U.S. and it's worse than you might think.

I plotted the time it took to retrieve 200 files from 1-200KB from two locations. The first was a server in the same city as me (kindly provided by SCT.com.au) and the second is with Slicehost in the US. We're comparing the effects of 15ms RTT vs. 250ms RTT on performance.

You might imagine the US server would simply add 250ms to the total download time. The reality is that it can take ten times longer to get files from the U.S. The TCP protocol includes provisions to avoid network congestion that were designed before we had broadband. I discussed these in my presentation 'Speed Matters' (slides here).

The Effects of Latency on Load Time

source data

The stepping you see is due to TCP Congestion Control (RFC 2581) and Delayed ACK (RFC 813).

My investigation was inspired by a talk by John Rauser at VelocityConf last month.

Speed Matters

Last week I lucky enough to attend O'Reilly Velocity 2010. In the heart of Silicon Valley, 1200 geeks assembled to discuss how we can make the web faster. We listened to techies from Amazon, Google, Yahoo, Facebook, Twitter, Microsoft, Firefox, Firebug, Chrome, and more talking about web performance and operations. The trip was worth it. Check out the videos.

Last night I had the opportunity to present a little of what I've learned about Web Performance to the Melbourne Ruby User Group. It's clear from feedback that we care about reducing page load times and most of us have used performance analysers like YSlow. Folks appreciated metrics from Google, Amazon and Wikia that show a small difference in page load time (100ms) affects sales, abandonment and conversions. This kind of thing helps us make a business case to employers and clients for speeding up the web.

Australia is a "special place" network wise. The insane cost of bandwidth and services means many of us choose cheap hosting in the U.S. even when the majority of our users are in Australia. This means choosing increasing latency from ~20ms to >200ms. Latency affects TCP performance even more than most of us would guess. I'm currently looking into just how big a difference this makes.

Which leads to the next question, where are the great VPS and DNS services in Aus? Suggestions appreciated!

View more presentations from mbailey.

Chrome is Most Popular Browser!

At least amongst visitors to my blog yesterday.

And no, I don't normally get 240 visitors to this blog but I'd just made an announcement that obviously appealed to people with good taste in browsers!

Asset Fingerprinting with Paperclip

Update: My Asset Fingerprinting patch was included in paperclip-2.3.4 in Oct 2010. Thanks jyurek!

One of the tricks to improving the performance of your website is to optimize caching. By instructing browsers and proxies to cache assets that don't change very often (css, images, javascript), we can reduce page load time as well as bandwidth costs.

By instructing browsers and proxies to never check for updated versions of cached assets we can further speed things up. Google recommend the use of fingerprinting to enable dynamic updating of cached content. This simply means including a fingerprint of the resource in it's URL so the URL gets updated when the resource changes.

Paperclip is a popular file attachment plugin for Ruby on Rails. It was a perfect candidate for fingerprinting support as it generates resource URLs from stored data.

Enabling it is as simple as:

1. Install paperclip-2.3.4
2. Add an :attachment_fingerprint column to your database
3. Include :fingerprint in the path for your attachment (see below)

has_attached_file :avatar,
  :styles => { :medium => "300x300>", :thumb => "100x100>" },
  :path => "users/:id/:attachment/:fingerprint-:style.:extension",
  :storage => :s3,
  :s3_headers => {'Expires' => 1.year.from_now.httpdate},
  :s3_credentials => "#{RAILS_ROOT}/config/s3.yml",
  :use_timestamp => false</code>

This enables us to set far future expire headers so that browsers don't need to check for a newer version. If a change does occur, say because a user uploads a new avatar, the new filename will be rendered in your html and the cached version will be ignored.

The example above will set Expires headers in S3. If you're using local storage you can configure your webserver to do something similar.

We disable the timestamped query string because some proxies refuse to cache items with query strings.

Fingerprinting support was added to paperclip-2.3.4 on 6 Oct 2010.

For more info on optimizing for caching:

http://code.google.com/speed/page-speed/docs/caching.html

Saving Pets, Risking Lives

[UPDATE] In response to feedback from friends I'm referring to the blogger as Rainbow Bear (Bo).

A week ago I wrote to the author of one of my favourite blogs with concerns about an image they had posted with the caption "A pit bull boxer cross'. This was just days after the Victorian Govt. announced it was setting up their 'Dob in a Dangerous Dog' hotline in preparation for a search and destroy mission for pit bulls and pit bull crosses based purely on appearance. The dog in the picture did not look like an American Pit Bull Terrier.  I asked that he remove the photo from his  blog because it may endanger similar looking dogs.

Hi Bo,

as you know, innocent family pets risk being seized and killed in Victoria due to their appearance. Dog without any 'pit bull' in them are already suffering because Authorized Officers think they resemble a pitbull cross.

You have published a photo with the caption "A pit bull boxer cross".

• How do you know the genetic origins of that dog?

• Given many dogs without 'pit bull' in may have a similar appearance to that dog, how can you justify putting them at risk by allowing it to remain on your site?

Please do not post pictures claiming to be 'pit bull crosses' as this may end up hurting dogs. Please also remove the one you've got up there.

I'm happy to discuss if you disagree but please don't continue to ignore my request for you to remove this image.

Mike Bailey

PO Box 2016 Fitzroy VIC 3065 mike@goodfordogs.org www.goodfordogs.org

A Week Later Rainbow Bear Responded, By Blog Post

A week passed without a response from Bo. Today I noticed he had put together a blog post on his blog defending their refusal to remove the image.

[caption id="attachment_408" align="alignleft" width="150" caption="Picture changed for privacy reasons"][/caption]

"I chose the image of a smiley, bright eyed, well-behaved dog, BECAUSE it was a smiley, bright eyed, well-behaved dog. We need to have more of these images to call upon, not less if we’re ever to overcome the ignorant, misguided aspersions generated by media hype and demonisation."

He then went on to put words in my mouth,

"Much better, they feel, that we hide all references to pit bulls for fear of making the situation worse."

Promoting Positive Images of Actual Pit Bulls

Posting positive images of American Pit Bull Terriers is a great idea. The difference is that sites promoting positive images of Pit Bulls tend to use dogs who at least look like APBTs.

The Victorian Govt is gearing up Council Officers to seize and destroy family pets based on appearance. They're saying they will declare dogs who fit the new id standard even if they're obviously not purebred APBTs. Don't put dogs without any pit bull in them in the firing line by posting photos claiming to be Pit Bull crosses.

Weigh Up The Risks and Benefits

It's naive to think you can change public perception when an issue is as hot as this. The media and government responded swiftly and in unison following the tragic death of Ayen Chol.

Conversely, posting images of cross bred dogs and saying "there's one" may well lead to people worrying about their own dog or dobbing in their neighbours. We're gearing up for the witch hunts over here.

Don't Give Council Workers More Ammo

Regardless of the genetic history of that dog, I don't believe posting photos of alleged Pit Bull crosses is responsible. It's not possible to determine the genetic history of a mixed breed dog by visual examination but this is what our Government is telling Council Officers to do. Don't give them more ammo.

Growing your Xen guest disks with xen-resize-guest

This morning I woke to a notification from nagios telling me the disk was full on one of the slices I manage for a client. I knew what I had to do. As we run Xen virtual servers on dedicated hardware I could shut down the slice, increase the lvm size, grow the filesystem and then fire up the slice again. What I didn't look forward to was reading man pages and  googling to find out exactly how to do that.

I was glad to discover that Steve Kemp's xen-tools-3.9 (a collection of scripts for creating and managing xen virtual machines) contains a command that does just what I needed. This script is not included in Ubuntu 7.10's package (xen-tools-3.5) so I just copied the script from the source distribution.

Xen-tools is well thought out, simple to use and easy to extend. I've been using it since I started using Xen and highly recommend it.

xen-tools-resize allowed me to grow the main disk for my gateway slices from 2GB to 10GB in about a minute with just one command. That left me with time to  tell you about Steve's awesome  toolkit! Steve's also got some great sysadmin tips at http://www.debian-administration.org/

    root@sm01:~# xen-resize-guest --hostname=gw1 --add=8Gb
    Preparing to resize image: /dev/vm_local/gw1-disk
    Sleeping for 9 seconds to allow cancel
    Sleeping for 8 seconds to allow cancel
    Sleeping for 7 seconds to allow cancel
    Sleeping for 6 seconds to allow cancel
    Sleeping for 5 seconds to allow cancel
    Sleeping for 4 seconds to allow cancel
    Sleeping for 3 seconds to allow cancel
    Sleeping for 2 seconds to allow cancel
    Sleeping for 1 seconds to allow cancel
    Sleeping for 0 seconds to allow cancel
    DO NOT INTERRUPT
      Extending logical volume gw1-disk to 10.00 GB
      Logical volume gw1-disk successfully resized
    Checking filesystem
    e2fsck 1.40.2 (12-Jul-2007)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/vm_local/gw1-disk: 22115/262144 files (1.7% non-contiguous), 264768/524288 blocks
    Resizing in place
    resize2fs 1.40.2 (12-Jul-2007)
    Resizing the filesystem on /dev/vm_local/gw1-disk to 2621440 (4k) blocks.
    The filesystem on /dev/vm_local/gw1-disk is now 2621440 blocks long.

    All done