Skip to content

commandline

BlinkSale - How To Say Goodbye

What do you do when you want to stop using a service but it's going to be a lot of effort to extract your data? Write a script to do it of course!

I've been using BlinkSale for my invoices since 2007 and it's served my needs well. Since being bought, BlinkSale has increased their monthly costs. In order to stop using them I needed to download 90 invoices as PDF's. The following script automated the process for me.

If you have Ruby and the mechanize rubygem you can download and run it with:

$ curl -L http://bit.ly/mkQqan -o blinksave && ruby blinksave

Because it asks for your BlinkSale password you really should read through the code to make sure this is not a phishing trip! Look how easy Mechanize makes this kind of task.

SSH compression can slow you down

Repost from my old blog

When copying some large files between hosts at our rack I noticed some pretty poor transfer speeds. Having recently forked out $70 for a rackmount gigabit switch I wondered what was slowing things.

It seems ssh was trying to help out by compressing everything however compressing the data took more than twice as long as transferring the uncompressed data. I proved this by disabling compression at the commandline and seeing the transfer speed more than triple.

mbailey@modu1:~/vm$ scp -r r03 modu4:vm/
Ubuntu 64-bit-f001.vmdk          7%  147MB   8.1MB/s   03:54 ETA

mbailey@modu1:~/vm$ scp -r -o 'Compression no'  r03 modu4:vm/
Ubuntu 64-bit-f001.vmdk        100% 2048MB  28.1MB/s   01:13 ETA

So how can we put this into practice everywhere?

Setting 'Compression no' in /etc/ssh/ssh_config will do the trick for me. There's no need for my vmware hosts to be compressing files I copy between them. I do want compression when copying files over shared/slower links but I can achieve that by initiating connections that benefit from compression on boxes that are configured to use it.

If I want to enable compression I can always use the '-C' switch to enable compression.

JSON from the Command Line

JSON is a beautiful format for storing objects as human readable text. It's succeeded where XML has failed. Not only is it not shit, it's actually quite good! But don't just take my word for it, have a look at some of the "cool" ways you can generate and consume JSON.

Syntax Highlighting for Vim

Useful when writing or editing JSON. Grab it here and drop into ~/.vim/plugin/

Consuming with CURL

Disable curl's progress bar and enable compression

$ echo "silent=true" >> ~/.curlrc
$ echo "compressed=true" >> ~/.curlrc

Install a Node.js package that will help us prettify output

npm install -g jsontool

Now let's grab a record from Facebook

$ curl graph.facebook.com/stoptheclock | json
{
  "about": "Abolish the 28 Day Rule for Victorian Shelters\n\nhttp://stoptheclock.com.au\n\ninfo@stoptheclock.com.au",
  "category": "Community",
  "founded": "2010",
  "is_published": true,
  "mission": "To bring an end to the law requiring Victorian shelters to kill healthy adoptable cats and dogs after four weeks.",
  "talking_about_count": 2,
  "username": "Stoptheclock",
  "website": "http://stoptheclock.com.au",
  "were_here_count": 0,
  "id": "167163086642552",
  "name": "Stop The Clock",
  "link": "http://www.facebook.com/Stoptheclock",
  "likes": 5515
}

Storing our JSON

The simplest solution is to save it to a file...

$ curl -s graph.facebook.com/stoptheclock | prettify_json.rb > fb_gfd.json

...but other applications can accept JSON such as CouchDB

Here we'll create a new database, store our JSON to it and then retrieve it.

# Create a database on CouchDB
$ curl localhost:5984/_all_dbs-X PUT localhost:5984/facebook
{"ok":true}

# Check it's there
$ curl localhost:5984/_all_dbs
["facebook","test"]

# Save our document to it
$ curl -X PUT localhost:5984/facebook/stoptheclock -d @fb_stc.json
{"ok":true,"id":"stoptheclock","rev":"1-f0422f8044e911b2f97c6ad71136eda1"}

# Check it's there
$ curl localhost:5984/facebook/stoptheclock | json
{
  "about": "Abolish the 28 Day Rule for Victorian Shelters\n\nhttp://stoptheclock.com.au\n\ninfo@stoptheclock.com.au",
  "category": "Community",
  "_rev": "1-f0422f8044e911b2f97c6ad71136eda1",
  "_id": "stoptheclock",
  "founded": "2010",
  "is_published": true,
  "mission": "To bring an end to the law requiring Victorian shelters to kill healthy adoptable cats and dogs after four weeks.",
  "talking_about_count": 2,
  "username": "Stoptheclock",
  "website": "http://stoptheclock.com.au",
  "were_here_count": 0,
  "id": "167163086642552",
  "name": "Stop The Clock",
  "link": "http://www.facebook.com/Stoptheclock",
  "likes": 5515
}

Querying CouchDB is outside the scope of this tutorial

How do I query the data now it's in CouchDB? How do I query the data now it's in CouchDB?

Posts in this series: