What do you do when you want to stop using a service but it’s going to be a lot of effort to extract your data? Write a script to do it of course!
I’ve been using BlinkSale for my invoices since 2007 and it’s served my needs well. Since being bought, BlinkSale has increased their monthly costs. In order to stop using them I needed to download 90 invoices as PDF’s. The following script automated the process for me.
If you have Ruby and the mechanize rubygem you can download and run it with:
curl -L http://bit.ly/mkQqan -o blinksave && ruby blinksaveBecause it asks for your BlinkSale password you really should read through the code to make sure this is not a phishing trip! Look how easy Mechanize makes this kind of task.
Have you stopped listening to new music? If you don’t have cool “music seeker” friends providing your aural fix then perhaps it’s time you become that person. If you’re into the shit I’m into then you’re in luck my friend. A fellow known only as “Shankly” has been putting monthly compilations on the net for a few years now. While much of it won’t take your fancy, some of it might and that can lead you to some great new music.
Sounds Great, What Do I Do
- Download this months edition of “Shanklys Mixtapes”
- Listen to the 80 tracks noting the ones you like
- Make a shortlist of albums to download. On a good month I find 5.
- After listening to the albums a few times, go buy the best ones.
Doesn’t That Make Me A Criminal?
Hey, how many albums did you buy last year? None? Because you don’t know what to get?
What’s criminal is that people don’t know this stuff is out there. Why does Radio and TV play absolute shit when there are true artists creating beautiful music.
What’s criminal is that no one told me how fucking amazing Mogwai is. Till I heard them on Shankly’s Mixtape for Jan 2011 that is.
April 2011
Mix 1
1. Vaccines – If You Wanna (Live)
2. REM – Alligator Aviator Autopilot Antimatter
3. Foo Fighters ΓΓé¼ΓÇ£ Rope
4. Arctic Monkeys – Don’t Sit Down Cause I’ve Moved Your Chair
5. Debbie Harry & Franz Ferdinand – Live Alone
6. Wombats – I’m A Robot Like You
7. Glasvegas – The World Is Yours
8. Elbow – Lippy Kids
9. Wolf Gang – Dancing With The Devil
10. White Lies – Is Love (Stereolab Remix)
11. Lykke Li – Rich Kids Blues
12. Yeasayer – Swallowing the Decibels
13. Uffie – Wordy Rappinghood (Evian Mix)
14. Cold War Kids – Mine Is Yours (Passion Pit Remix)
15. Kassidy – I Dont Know
16. Deftones – Drive (Cars)
17. Gruff Rhys – Take A Sentence
18. Sara Lov – Hold Me Now (Thompson Twins)
19. Aretha Franklin – God Bless The Child
20. Radiohead – Supercollider
Mix 2
1. Decemberists – Down By The Water (Live)
2. Okkervil River – Rider
3. My Morning Jacket – Holdin On To Black Metal
4. New Pornographers – A Drug Deal Of The Heart
5. Laura Stevenson & The Cans – Master of Art
6. Felice Brothers – Fire at the Pageant
7. Zoey Van Goey – You Told The Drunks I Knew Karate
8. Iron & Wine – Godless Brother In Love
9. National – Exile Vilify
10. Antlers – Every Night My Teeth Are Falling Out
11. Boxer Rebellion – Step Out of the Car
12. Cascadeur – Walker
13. Watson Twins – Angelene
14. Low – Especially Me
15. Caroline – Sleep
16. William Fitzsimmons – The Tide Pulls From The Moon
17. King Creosote & Jon Hopkins – Bubble
18. Roddy Woomble – Work Like You Can
19. Matt Sweeny & Bonnie Prince Billy – Must Be Blind
20. Shearwater – Missing Islands
Mix 3
1. Akron Family – Island
2. Dengue Fever – Cannibal Courtship
3. Dirtbombs – Good Life (Inner City)
4. Grinderman – Evil
5. PJ Harvey ΓΓé¼ΓÇ£ England
6. Bill Callahan – America!
7. Parenthetical Girls – Handsome Devil (Smiths)
8. British Sea Power – Stunde Null
9. Art Brut – Clever Clever Jazz
10. My Dad Vs. Yours – Wildcat Strike
11. Buffalo Tom – Down
12. Flaming Lips – Drug Chart
13. Airborne Toxic Event – All at Once
14. Parts And Labour – Fake Names
15. Help Stamp Out Loneliness – Record Shop
16. Thomas Tantrum – Its Like Drums
17. Snowmen ΓΓé¼ΓÇ£ Hyena
18. Young Knives – Love My Name
19. Black Feather – Paperhouse (Can)
20. Ema – California
Mix 4
1. Dananananayckroyd ΓΓé¼ΓÇ£ Reboot
2. And So I Watch You From Afar – BEAUTIFULUNIVERSEMASTERCHAMPION
3. Standard Fare ΓΓé¼ΓÇ£ Suitcase
4. Pains Of Being Pure At Heart – Girl of 1,000 Dreams
5. Boogie Monster – Castle In The Clouds
6. Los Campesinos! – Light Leaves, Dark Sees
7. Woebot – Blues
8. Mogwai – Hasenheide (B)
9. Sorry Kisses – Drifting In the Dark
10. Four Tet ΓΓé¼ΓÇ£ Pinnacles
11. Johnny Foreigner – What Drummers Get
12. Tinklers – Don’t Put Your Finger In The Fan
13. Grails – All The Colors Of The Dark
14. Mazes – Most Days
15. Battles ΓΓé¼ΓÇ£ Futura
16. Bearsuit – Princess, You’re a Test
17. She’s Hit – RE_PEATER
18. Moonbell – Nostalgia For The Future
19. Lovely Eggs – Muhammad Ali and All His Friends
20. Kontakte – The Ocean Between You And Me
slicehost - Built for Developers
When Rackspace bought Slicehost in 2008 I suspected it would only be a matter of time before they ruined it. In May 2011 Rackspace announced that they will be shutting down Slicehost. It’s sad news for a lot of people who have been a part of the “slicehost community” for this was a company that customers felt a part of. Co-founder Matt Tanase says in this video interview the news “tears him up inside”.
When I first signed up for a slicehost VPS in 2006 I was dubious about how $20/month could buy me a server with 256MB memory and 100GB of traffic throughput. There was no signup fee, no minimum contract period and the management software was so simple and clear. My concerns that it would be oversubscribed and poorly supported were proved wrong. I was soon won over!
The guys who ran the company, Matt and Jason, also ran the servers. They chatted with customers on IRC and answered emails. Somehow they even found time to produce a podcast where they spoke candidly about their experiences and plans. Through the podcast the guys really made us feel like we were there with them.
One of the hot topics in the early days was the waitlist. Slicehost was an overnight success and the guys had to deal with its popularity. The waiting list for new VPS’s was stretching out to weeks when they came up with a novel solution. If you were willing to prepay for future months service you could move further up the queue. I can recall when paying for 6 months in advance would give you a server same day. It solved their cash flow problem and allowed them to buy more hardware.
The following year they explained the downside to this approach. The tax office saw all the prepayments as income and wanted them to pay tax on them but saw their expenditure on servers as capital expenses which could only be depreciated over several years. They were faced with a big tax bill.
I’m sad to see slicehost being dismantled. This comes on the tail of Cisco announcing their plans to kill the Flip. What is it with companies buying great products and killing them?
I’ll leave you with episode 4 of the slicehost podcast. I don’t think you’re ever going to get something this raw from Rackspace.
Cisco bought Flip camcorder for $580 million and is closing it down
Use Virtualization
If I could offer you only one tip for the future, virtualization would be it. The long-term benefits of virtualization have been proved by scientists, whereas the rest of my advice has no basis more reliable than my own meandering experience.
OK, so which server OS virtualization platform are we going to use in our home data center? We’ve got a few choices:
VMware’s vSphere Hypervisor
VMware shows little regard for non Windows folk, requiring us to run the vSphere Client under a virtual machine. running windows. VMware can fuck right off.
Xen
I love Xen and am pleased to see debian is again supporting it. Amazon use Xen for their EC2 service and the last time I looked the majority of big virtual hosting providers were using it. A couple of years back I wrote a web interface for Xen called Xenium.
After all that gushing you’re probably going to think I’m recommending you use Xen right? Nope!
Oracle VirtualBox
I think my love for Xen caused me to ignore this option until recently.
VirtualBox is cross platform open source (GPL2) virtualization software for Linux, OSX, Windows and Solaris. It’s easy to install and someone has created a web frontend (phpvirtualbox) for it that looks just like the native GUI.
I’ve only just started exploring VirtualBox but for ease of installation, freedom and functionality it looks like a winner for hosting @ home.
Things I like about VirtualBox
- sound from Windows guest is playing on my Linux host
- mouse moves smoothly with no need to use keys to release
- easy to share a directory from host to guest (well…windows needed reboot)
- VRDP gives you RDP even when your OS doesn’t
My Billion 7300 modem/router does a simple task well.
My first step on the path to hosting @ home was to get a good Net connection. I selected a Naked ADSL plan from Australian provider Internode. Australians pay crazy high prices for Internet but $60 a month for 150GB (combined download/upload) doesn’t seem too steep. In order to host @ home I needed to disable the Internode network firewall to enable incoming traffic.
Internode gives me a dynamic IP (which I actually prefer). Most DSL routers come with support for dynamic DNS built in and mine does a great job of updating my DNS entry within seconds of my IP changing. All other domains hosted here will have CNAMEs pointing this host only a single hostname needs to be updated when my IP changes.
When my modem light glowed steady on Friday I knew I could get started configuring the router. I’m fond of Billion modem/routers which sell for around $60.
Configuring the router
ADSL Routers tend to be pretty easy to configure via their web interface providing you remember the admin password or have something to poke in the hole to reset it to factory defaults. The other thing you have to work out is the IP address the modem is running on. For some reason 192.168.0.1 is not the universal standard – my modem was on 192.168.1.254. Go figure. Here’s what I do when setting up a new host@home network.
- Change root password from the factory default
- Configure DHCP to handout my ISP’s nameservers and my own domain
- Configure DHCP to IPs from 100 – 200 (I reserve others for manual addressing)
- Forward incoming connections to a gateway IP (which forwards traffic using HAProxy)
- Configure dynamic dns
My Billion ADSL router updates my dynamic dns entry when my IP changes.
I installed apache2 on my gateway host to test external access. You should be able to access it here: home.failmode.com
In our next installment…
In my next post I’ll cover setting up HAProxy on the gateway host to so that incoming requests can be routed to the correct internal servers.
[Repost from my old blog]
When copying some large files between hosts at our rack I noticed some pretty poor transfer speeds. Having recently forked out $70 for a rack mount gigabit switch I wondered what was slowing things.
It seems ssh was trying to help out by compressing everything however compressing the data took more than twice as long as transferring the uncompressed data. I proved this by disabling compression at the commandline and seeing the transfer speed more than triple.
mbailey@modu1:~/vm$ scp -r r03 modu4:vm/ Ubuntu 64-bit-f001.vmdk 7% 147MB 8.1MB/s 03:54 ETA mbailey@modu1:~/vm$ scp -r -o 'Compression no' r03 modu4:vm/ Ubuntu 64-bit-f001.vmdk 100% 2048MB 28.1MB/s 01:13 ETA
So how can we put this into practice everywhere?
Setting ‘Compression no’ in /etc/ssh/ssh_config will do the trick for me. There’s no need for my vmware hosts to be compressing files I copy between them. I do want compression when copying files over shared/slower links but I can achieve that by initiating connections that benefit from compression on boxes that are configured to use it.
If I want to enable compression I can always use the ‘-C’ switch to enable compression.
Humans, like the apes we evolved from, are social creatures. Cooperation has given survival advantages to our species. One interesting aspect of our wiring is empathy. It’s been argued that this involves mirror neurons, which fire both when an animal acts and when the animal observes the same action performed by another. Whatever the cause, we feel other peoples feelings and therefore share in the benefits when we make someone feel better. Conversely, causing someone distress should give us unpleasant feelings. This seems like a simple yet effective system for social regulation that doesn’t require modern (human) innovations like verbal language, law or religion.
So how can people still be cruel to other people if they’re going to feel their emotions? Did we evolve a way to selectively shutdown this mechanism? If your tribe was under attack, surely it would be advantageous if your warriors could suppress empathic responses just long enough to beat your enemies to a pulp. You would want them to be re-enabled once the threat has gone though!
I think we have built in empathy circuit breakers that get activated when we see people we dislike. There may be other ways to suppress empathy such as “just following orders” but it’s the identification of people as bad or an enemy that has me curious. Is this a major contributor to cruelty and callousness from otherwise kind individuals? Do we become ‘selectively psychopathic’ by shutting down an important part our ‘humanity’?
Some advice I always had trouble comprehending was “love your enemies“. It may turn out to be another way of saying, “don’t disable your empathic mechanisms”. While this may not be helpful advice for a soldier on a battlefield, it’s probably good advice for us in our schools, workplaces and social gatherings.
I suspect a way to achieve this is not view anyone you deal with as ‘bad guys’ which reminds me of another quote attributed to Jesus, “Do not judge others“.
There seems to be a fair bit of research going into modulation of empathic responses. I suspect there’s a significant difference between the response of males and females to seeing a person being punished for wrongdoing. In the TV show The Shield, it’s made very clear to you who the bad guy is in each episode and he usually gets a pounding by the end of it. My dad used to enjoy shows like The Professionals when I was a kid but my mum found the violence distressing. I wonder whether males, being the ones who probably had to defend the camp, developed the ability to ‘reverse the polarity’ on their empathic circuits and actually gain pleasure from seeing ‘bad guys’ suffer?
Perhaps avoiding the human tendency to judge others unfavourably can help us avoid triggering a survival mechanism which doesn’t have a place in modern society and in doing so, allow us to act better toward one another.
Jesus must have given this a lot of thought.
Yes, the misspelling in the title is deliberate. See image below.
If you’re building expensive tools for managing server infrastructure, please don’t assume your users are idiots. Warning me about the possible consequences of my action is fine. Preventing me from doing what I want to do won’t endear me to your product and poor spelling won’t increase my confidence. I’m LOOSING confidence in you VMware.
"You can't go there. Sorry, it's the rules."
Rob Postill had a little rant about VMware last night.
Web browsers don’t tend to display JSON very nicely. See how yours does with this example. Some handy plugins that help with this. While they currently only allow viewing, how long before we se some PUT/POST action?
JSONView for Firefox
Normally when encountering a JSON document (content type “application/json”), Firefox simply prompts you to download the file. With the JSONView extension, JSON documents are shown in the browser similar to how XML documents are shown. The document is formatted, highlighted, and arrays and objects can be collapsed. Even if the JSON document contains errors, JSONView will still show the raw text.
There is also an Unofficial JSONView port for Chrome and the Unfortunately Named “XML View” for Safari
When installed, your JSON will look more like this:
JSONView browser plugins for Firefox/Chrome makes JSON pretty
Posts in this series
JSON is a beautiful format for storing objects as human readable text. It’s succeeded where XML has failed. Not only is it not shit, it’s actually quite good! But don’t just take my word for it, have a look at some of the “cool” ways you can generate and consume JSON.
Ruby support for JSON
Ruby’s JSON library makes parsing and generating JSON simple. You can run the following in an irb session
# Ruby converts from hash <--> json
irb(main):001:0> require 'json'
irb(main):002:0> json_text = { :name => 'Mike', :age => 70 }.to_json
=> "{\"age\":70,\"name\":\"Mike\"}"
irb(main):004:0> JSON.parse(json_text)
=> {"name"=>"Mike", "age"=>70}
HTTParty helps with communicating with RESTful services
Here we grab a record from Facebook
# Retrieve a JSON Resource require 'pp' require 'json' require 'httparty' pp JSON.parse HTTParty.get('https://graph.facebook.com/goodfordogs').response.body {"name"=>"GoodForDogs", "category"=>"Local business", "username"=>"Goodfordogs", "id"=>"171644807551", "website"=>"http://Goodfordogs.org", "link"=>"http://www.facebook.com/Goodfordogs", "likes"=>871, "picture"=> "http://profile.ak.fbcdn.net/hprofile-ak-snc4/hs235.ash2/50503_171644807551_2269219_s.jpg"}
HTTParty gets Classy
Creating a simple class allows you to DRY things up a bit
require 'httparty' class Facebook include HTTParty base_uri 'https://graph.facebook.com/' # default_params :output => 'json' format :json def self.object(id) get "/14063940" end end require 'pp' pp Facebook.object('goodfordogs').parsed_response {"name"=>"GoodForDogs", "category"=>"Local business", "username"=>"Goodfordogs", "id"=>"171644807551", "website"=>"http://Goodfordogs.org", "link"=>"http://www.facebook.com/Goodfordogs", "likes"=>871, "picture"=> "http://profile.ak.fbcdn.net/hprofile-ak-snc4/hs235.ash2/50503_171644807551_2269219_s.jpg"}
Rails support for JSON
ActiveSupport::JSON knows how to convert ActiveRecord objects (and more) to JSON. Simone Carletti explains how this differs from the standard lib.
## Encode json = ActiveSupport::JSON.encode(object) # extra methods like :include json = Offering.first.to_json(:include => :outlet, :methods => [:days_waiting]) ## Decode ActiveSupport::JSON.decode(json)
Rails3 niceness
Adding JSON to your Rails3 app doesn’t require a lot of extra code. You can specify method calls and associated objects to include as well as restrict the attributes returned. Simple eh?
class PostController < ApplicationController respond_to :json, :html, :jpg, :xml def index respond_with(@posts = Post.all), :methods => [:average_rating], :include => :comments end def show respond_with(@post = Post.find(params[:id])), :only => [:name, :body] end end
Posts in this series
