Category: Code

  • Setting up a new mac

    Setting up a new mac

    I bought a new iMac recently and figured it would be worthwhile to write down everything I did to get it to where I wanted it.

    • Install bootstraps:
      • Chrome – Preferred browser
      • Iterm2 – Terminal
      • Homebrew – The missing package manager
      • Atom – Text editor for code
    • Install brew packages and ruby
    • Configure git
    • Copy across SSH keys (Airdrop is a good way to do this)
    • Configure SSHD for passwordless entry
      • Modify /private/etc/sshd_config
      • Put the SSH key in ~/.ssh/authorized_keys
    • Sync remote files from old machine to new with rsync (Not everything is in Google Drive / Dropbox)
    • Install from the app store
    • Install from web
      • Backblaze – Whole mac offsite backup
      • Dropbox – Sync work files
      • Google Drive – Sync personal non-photo files.
      • Google Photo Sync – Yes really there are two different google clients
      • Emacs – Text editor for when you need to automate stuff
      • Gimp – Image editing
      • Office – Hey, I still need Excel
      • Skype – Talking to mom
      • VMWare – Run Windows and Ubuntu in a box
      • Paparazzi – Great screenshots
    • System Preferences
      • Dock
        • Set size as small as it will go, turn on zoom
      • Security
        • Turn on the firewall
      • Sharing
        • Permit remote login
      • Mouse
        • Turn on right click gesture
    • Remove from the dock:
      • Safari
      • Mail
      • Contacts
      • Calendar
      • Notes
      • Todo
      • Pages
      • Numbers
      • iTunes
      • iBooks
  • Unevenly spaced time series analysis

    Unevenly spaced time series analysis

    A number of years ago I bought a Withings connected scale. It’s great. You get on every morning, it logs your weight and uploads it to the internet where it can be analyzed.

    Recently, I decided to use the Withing API to build a plot that I could include on my homepage to keep an eye on my weight. Something every weight-watcher knows though is your weight fluctuates and so it’s great to keep an average. Unfortunately, unevenly spaced time series do not interact well with simple moving averages, so I found this paper and converted the code into javascript:

    The algorithm expects data in a variable called json_data. It’s an array of arrays, each of which is a timestamp and data pair. (This is the form that Highcharts expects.) The key factor is tau, the length of the time window over which you’re averaging. I have it set to a week here which I think is about right for looking at human weight. (3.5 days works quite well too.)

    The results are here:



    The code in context and a cleaner running example are both available. You can also see the raw data I use to fuel the graph.

  • Keyboard modified clicks

    In case you’re ever wondering, here is a full list of the “key modified clicks” that the major browsers look for:

    (Note that on Windows, “cmd” is “ctrl”)

    Chrome, Firefox:
    shift: new active window
    cmd: new tab
    cmd-shift: new active tab

    Safari:
    cmd: new tab
    cmd-shift: new active tab
    cmd-opt: new window
    cmd-opt-shift: new active window

    IE:
    shift: new active window
    ctrl-shift: new active tab

    Note that you also need to deal with right clicks and “open in new window”, “open in new tab”, etc….

  • Wow Google Spreadsheets

    I’ve long been a fan of Google Spreadsheet. Like all disruptive technologies, it doesn’t meet the needs of most customers of the incumbent product, Excel. I’m no banker, but even my relatively neophyte hands keep trying to hit F4 to lock a cell reference or hit F2 to edit a cell. I miss being able to right click to format a cell, the lag time of javascript sometimes annoys the crap out of me, etc… I would not try to convince a Wall Street number jockey to switch from Excel just yet. No way.

    BUT, also like all disruptive technologies Google Spreadsheet is quietly getting good at things that Excel can only dream of. Since it launched, Google Spreadsheet has had great sharing and concurrent editing capabilities. A team at school used a shared Google Spreadsheet as a “factory information system” in a simulated factory exercise and it was amazingly powerful.

    More recently, Google has added amazing data extraction techniques. Put =GoogleFinance(“Oil”, “Price”) in a cell and you’ll get the price of oil updated in realtime. The following will pull all the headlines off Techmeme: =importxml(“http://www.techmeme.com”, “/html/body/div[2]/div/div[4]/div/div[2]/div/div/div/strong/a”)

    There are also functions for trolling arbitrary HTML, for parsing RSS and Atom feeds and for importing CSS feeds.

    Where the mind really starts to boggle is when you think about how every cell in every google spreadsheet effectively has an URL. So it should be almost as easy to publish from your sheet as it is to pull in from somebody else’s. Suddenly, there are network effects for spreadsheets, where your work can leverage off the work of others. This ability to ref a cell in someone else’s sheet doesn’t seem to be enabled just yet, but I can’t imagine Google isn’t working on it. I’m sure there are some issues around permissioning and detection of dependency loops.

    Throw in a little work on keyboard shortcuts and a little Google Gears magic to make the app more responsive and workable offline and Excel will be starting to feel the heat…

  • PicasaWeb Downloader

    In the genuinely useful software dept., here is a little app written in C# (source included!) that will download an entire google web album. Great for grabbing copies of all of your friends photos.

  • Theme weirdness

    I upgraded WordPress to the latest version and my theme seemed to be interacting poorly with the new Schema. So I’ve reverted to the default theme and I’ll find a new one I like at some point soon.

  • Computing distances with Lat/Lon

    Have you ever wanted to compute distances with latitude and longitude? At first it seems so simple. We all learned the pythagorean theorem in elementary school:
    d=sqrt((x2 – x1)^2 + (y2 – y1)^2)
    So if we have our two points (x1,y1) and (x2,y2) then we should be able to find (x2-x1,y2-y1) and find the distance using the Pythagorean theorem, right? Well, no.

    The Pythagorean theorem finds the straight line distance between two points. However, for two points any appreciable distance from each other on earth, trying to get to your destination would involve a lot of digging. Assuming you want to travel overland, you must travel further than the linear distance to account for the curvature of the Earth.

    A little further research yields the Haversine Formula which is useful for computing surface distances on a sphere and correctly accounts for points that are very close together:

    Haversine Formula (from R.W. Sinnott, “Virtues of the Haversine”, Sky and Telescope, vol. 68, no. 2, 1984, p. 159):

    dlon = lon2 – lon1
    dlat = lat2 – lat1
    a = sin2(dlat/2) + cos(lat1) * cos(lat2) * sin2(dlon/2)
    c = 2 * arcsin(min(1,sqrt(a)))
    d = R * c

    But what is R, the radius of the Earth?

    It turns out the Earth is not a sphere after all. The spinning motion compresses the planet so that the equitorial radius is considerably larger than the polar radius. The shape of the Earth is well approximated by an oblate spheroid with a polar radius of 6357 km and an equatorial radius of 6378 km.

    R’ = a * (1 – e2) / (1 – e2 * sin2(lat))^(3/2)

    where a is the equatorial radius, b is the polar radius, and
    e is the eccentricity of the ellipsoid = (1 – b2/a2)^(1/2).

    Wow! So there you have it.

    Of course if you’re trying to compute these values in client side javascript, it may take a while. It turns out that for short-ish distances at latitutudes within the continental United States, just pretending the Earth is a sphere yields errors of roughly 20 meters. That’s good enough for me..

    (All from Google and http://www.faqs.org/faqs/geography/infosystems-faq/ )

  • KML in Google Maps and the challenge of standards

    The Google Maps API is generally a pleasure to work with. After more or less inventing the whole Web 2.0 thing, the people behind Google Maps have continued to innovate, recently adding a Geocoder to their API, allowing address information to be translated into coordinates in Latitude and Longitude. Naturally having this information expands considerably the number of things you can do with Google Maps. Even better, it doesn’t just return Coordinates for an address, it “normalizes” a query and returns the address in a structured way.
    There’s no doubt that creating a global XML standard for addresses must have been a challenging thing to try and do. Some countries use street names in addresses others do not. Some nations use island names, others have no islands. Some (like America) has county names but do not use them in addresses. So Kudos to OASIS for creating a standard called xAL and kudos to Google for trying to use it in their Google Maps API. Unfortunately, the devil is always in the details.

    (more…)

  • The joys of screenscraping

    A while back, I ran across a great HackDiary entry extolling the virtues of using TagSoup and XPATH to do screenscraping from the web. TagSoup is a library that coerces all the ugly nasty HTML you find out in the wild into well-formed (although not necessarily valid) XML. While there’s no guarantee that the results are semantically the same as the input, it lets you use all your nice XML tools like XPATH to extract data. The entry does a great job of showing you how to use TagSoup with Xalan. However, the JDK has been updated with it’s own XPATH parser so it’s no longer necessary to import the Xalan library. Below is a code sample for using TagSoup and the default XPATH parser to retrieve the stock price of Google from Google Finance. Note that the whole “MutableNamespaceContext” implementation is just a workaround for a missing JDK method as documented in JDK Bug 5101859. If that bug gets fixed, the code could be simplified substantially.

    The usual disclaimers apply about this all being sample quality code. All error handling has been punted to keep the example length short, but you’d never really want to do that. Also, I’m having trouble preserving indenting in this HTML View, I’ll work on that. This code assumes JDK 1.5. Click through for the code itself.

    Does this code support the argument that Java is WAY too verbose? Absolutely.
    (more…)

  • Postfix

    My rimu host hosts a number of domains, including oroup.com, openrelay.com, cadabraco.com and drugmaps.com. I want to be able to send and receive email to those domains, but (at least for now) I don’t really want to set up dovecot locally and actually deal with mail. So the ideal thing is just to forward your mail to another email address. Postfix is a little daunting at first, but the key instructions turn out to be quite simple.

    Presto, instant email!