≡ Menu

ipython notebook tips and tricks for science research

David Lagattuta and I gave a seminar at CAS about using python, ipython and the ipython notebook. At the end of it we made 3 of our ipython notebooks available to all.

These notebooks have embedded in them examples of images, text, LaTeX, and even embedded YouTube screencasts that explain aspects of the notebook within the notebook itself.

  1. ipython notebook: Future of Sciencehttp://nbviewer.ipython.org/5742826
  2. Clean Code; Clear Codehttp://nbviewer.ipython.org/5742829
  3. The Future of Science — EXTRAShttp://nbviewer.ipython.org/5742830

Here’s a screencast that we embedded which shows some neat ipython notebook features:

As always, I’m looking to improve my coding (and presentation) skills, so constructive criticism is requested.



Hidden Universe IMAX 3D

I’ve been involved (http://hiddenuniversemovie.com/the-film/the-astronomers/) with the production of a 3D IMAX film called Hidden Universe.

The locations it’ll be showing (so far) are here: http://hiddenuniversemovie.com/theatre-locations/

Some photos that I took of the filming in Chile (back in November 2012).

2012-11 IMAX


Enhanced by Zemanta

Eating differently

I’ve been inspired to write up a bit about what I have been eating. The long and the short of it is that I’m basically following the “Paleo”/”Primal” diet (since July 2012), for health reasons (as in my goal was not for weight loss).

It breaks down into two lists: the “allowed/encouraged” and “not allowed”. Here’s how I play it:


  • fish
  • meat
  • fowl
  • eggs
  • bacon
  • avocados
  • greens (love me some spinach!)
  • fruits
  • vegetables (bell peppers, zucchini, onions, etc.)
  • nuts (almonds, walnuts, pine nuts)
  • oil (olive oil, avocado oils, coconut oil)
  • vinegars (balsamic, etc.)
  • tea


  • no grains (flour, wheat, cereals, etc.)
  • no potatoes
  • no legumes (peanuts, soy)
  • no beans
  • no corn
  • no refined sugar
  • semi-restricted dairy (butter and cheese are fine for me)

A small rant about the name. Paleo is short for paleolithic, as in the time period. It is also referred to as “caveman” diet and probably a few other names. I dislike this name because I find it misleading and unhelpful. The reason I do or do not eat the above things is that I’ve found it to be healthy to eat (or avoid) the above lists of food. That’s it. Not for any other reason that’s justified because “we evolved to eat this way” or other such claims. I do suspect that there’s something to the idea that grains (not just gluten) can cause reactions in some people like an inflammation response.

I don’t want to go into the details, but I just want to say that I’ve found that following these rules has been beneficial to me. As a scientist, I insist that my anecdote isn’t conclusive evidence, but it might lead to some interesting future experiments.

Also note that I decided to follow this diet after watching the following TED talk and reading a number of resources (which didn’t all agree). I decided that at the worst the diet was healthy (and likely healthier than I was eating at the time). At best, it could improve my health.

Free resources


TEDxIowaCity Dr. Terry Wahls

Your mileage may vary/buyer beware/consult doctor before changing stuff.

Enhanced by Zemanta

Data Reduction of VLT-UVES

The end goal will be to run UVES_headsort to correctly setup the reduction scripts to reduce science exposures. Let’s look at reducing the VLT-UVES science exposures for a single object: vesta.

Start with a bunch of raw UVES data files in a folder (let’s call it raw) that are named things like:

In this mess of files, there exist all manner of calibration and science exposures for the spectrograph in different settings. It’s a mess, so the first order of business is to organize this into something that can be used. It isn’t pretty, but here’s how I do this first step:

# I'm going to make this into a proper script one day, but here it is as it currently stands
# Replace OBJECTNAME with the object name in the exposures
ls /raw/UVES.2010*.fits > raw.list.all.1
dfits `cat raw.list.all.1` | fitsort DPR.TYPE DPR.CATG EXPTIME OBJECT INS.MODE DPR.TECH | \
grep -v "SLIT" | grep -v "CDALIGN" | grep -v "SimCal" | grep -v "DARK" | grep -v "DFLAT" | grep -v "TEST" | grep -v "FIBRE" | \
awk 'NR > 1 {if ($3!="SCIENCE" || ($3=="SCIENCE" && $4>0.1)) print $0}' | \
awk '{if ($4>=300.0 || $3!="SCIENCE" || ($4<300.0 && $3=="SCIENCE" && ($5!="OBJECTNAME")) || (($3=="CALIB" || $3=="TEST") && ($6!="ECHELLE,ABSORPTION-CELL" || $7!="ECHELLE,ABSORPTION-CELL"))) print $0}' | \
awk '{print $1}' > raw.list.1
grep -f raw.list.1 -v raw.list.all.1 > raw.list.exclude.1

What should happen at this point is a file is made: raw.list.1 which has all of the science and their associated files that need to be reduced. An important point here — you have to use the full path names.

(I’ve never used the helpreduce.py script at this stage, so you can try it (let me know how it goes), or just use the following command).

$ UVES_headsort raw.list.1 -info raw.info -list -c 45 45 -d

If the object name is correctly filled out in the science exposures (a genuine “if”), you’ll have a file named vesta.list which looks like:

$ head vesta.list 

I’ve written a really simple python script that helps set up the reduction scripts — it’s not well-commented but it’s also not too complicated.

# 2011-11-16 helpreduce.py
import sys
import os
import subprocess
import argparse
parser = argparse.ArgumentParser(description="filename")
parser.add_argument('infile', action='store', type=str, help='grabs commandline filename')
args = parser.parse_args()
stem = args.infile
if args.infile.split('.')[-1] == 'list':
  stem = args.infile.split('.')[0]
basecommand = 'UVES_headsort ' +  stem + '.list -info ' + stem + '.info -c '
hours = 0
warning = 10
maxhours = 200
while warning > 5 and hours < maxhours:
  hours += 1
  linein = basecommand + str(hours) + ' ' + str(hours) + ' -d'
  proc = subprocess.Popen(shlex.split(linein), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  warning = len(proc.communicate()[0].split('\n'))
  print "Warnings: ", warning, "Hours: ", hours
if hours > maxhours:
print "Correct hours: ", hours
print "Final command: ", basecommand + str(hours) + ' ' + str(hours)
commandfile = open(stem + '.command','w')
print "Making command file."
print >>commandfile, basecommand + str(hours) + ' ' + str(hours)

The basic usage:

$ helpreduce.py vesta.list

It will run UVES_headsort in debug mode with an increasing range of hours around the exposures until it includes the files it needs — it will crap out if you don’t have the right (or enough) calibration files. When it runs properly, this script at the end produces two files: vesta.info and vesta.command. Run

$ source vesta.command

and you should have a directory vesta/ which, if you cd into it should reduction scripts named things like:

$ cd vesta; ls reduce*

If you’re feeling lucky, you can run:

$ source reduce_master.cpl

and it will go through all of the reduction scripts one at a time and do all of the magic.


There’s still plenty of time to be a genius!

Stern-Gerlach Experiment de

Image via Wikipedia

There was an interesting article that showed up in many places that had some neat information in it. It also didn’t really explore one of the things that struck me as interesting.

The article(s) is MSN: The stroke of genius strikes later in life today. It talks about the idea that scientists do their best work before the age of 30.

It turns out that it definitely USED to be generally true; but it has increasingly not been the case. It turns out that for physics at least, 48 is the average age at which their most genius work is done.


One thing that struck me was this sentence:
“In fact, in 1923, the proportion of physicists who did their breakthrough work by age 30 peaked at 31 percent. Those who did their best work by age 40 peaked in 1934 at 78 percent.”

Let’s keep those numbers in mind for a minute. Consider for a second that it was a certain generation of physicists who bring about the Quantum Mechanics revolution, and not a particular age. Let’s consider the generation born in 1893-4.

If this generation was in the proper place due to an accident of birth, they might have the lion’s share of substantial breakthroughs over a twenty year span or so. This group of physicists would have turned 30 in 1923-24. This same group of physicists would have turned 40 in 1933-34.

So, it’s the same generation in both cases, the only thing that differed was when their “best work” was considered to have occurred. In fact, since the percentages peaked in both decades, it appears that that generation is the one that dominated physics for twenty years.

Enhanced by Zemanta
{ 1 comment }

Suggestions for Moving to Melbourne

Melbourne Docklands area, taken from the terra...

Image via Wikipedia

Now that I’ve been here just over a month, I decided to write down a few of the suggestions that I had about relocating to Melbourne while it is still relatively fresh in my head.


Step 0. Take out Aussie $$ via ATMs from your US $$ accounts.

Step 1. Get an Aussie bank account. It’s dead simple, just do it. Walk into a bank and do it.

Step 2. Get a mobile (cell) phone. For some reason, it seemed to me that everyone in Australia will conduct interactions via phone calls and not via email almost on pain of death. Perhaps it was just the specific things that I was trying to complete, but it seemed that no one wanted to email. This lead to a number of phone-tag back-and-forths, including one situation where a real estate agent and I called and missed each other 7 times over a span of about a week.


If you are looking to rent: I recommend going with www.domain.com.au and www.realestate.com.au websites. The process for finding an apartment is roughly as follows. Before you are allowed to apply for an apartment you have to physically inspect the apartment. The real estate agent posts a time on one of the websites I listed above that everyone interested in an apartment needs to arrive. They will advertise a place and say: on 17/09 12:30-1PM (remember they do the date day/month/year) — you need to show up at 12:30, maybe even earlier. They don’t usually stay the whole time. Also, the walkthroughs were fairly quick in my experience. When you show up, they will write your name and mobile phone number.

If you are looking for house shares try: melbourne.gumtree.com.au and swinburne.studystays.com.au.

Also, the addresses are listed like this: “343/57-59 Main Street” means: Apartment Number 343, Street Address 57 through 59. Don’t ask me, it just means you can show up at 57, or 59 and you’re at the same place either way.

The rent is listed as $/week — but you normally pay per month. (And it can be 4 or 4 1/3 times the weekly amount listed, so clarify w/ renter).


I applied to a number of places. Here is what I did and some suggestions.

If I noticed that a certain real estate company is representing a number of apartments that I am interested in looking at, I filled out their application with information that would be the same each application: the references, my name, my contact info, etc. I then made a “master copy” that I made photocopies. This allowed me to quickly fill out their application for several places with the same real estate agency.

Rent application credentials.

  • 1 page cv/resume; if you have a phd make sure you write: Dr. Jonathan Whitmore and in general during this application process, put Dr. wherever you can (or PhD student if that’s what you are, don’t lie). You are trying to be as impressive as can be.
  • Clear photocopies of as many forms of ID that you can find: passports, drivers licenses, staff ID cards, etc.
  • Include your letter of offer for employment
  • Include work visa (mine was the email printout from the Aussie government saying the terms).

Make a number of copies of your credentials so that you can easily fill out the application and attach your credential packet. I recommend filling out the applications for each of the apartments that you are going to look at before you go. This allows you to hand the entire filled out application packet if you are interested applying for the place.

Some final thoughts: weekends are busy and usually have many people looking at once. You have a better shot with during the week ones (less people seeing/applying means your odds are slightly better). Be friendly with the real estate agents. Finally, try to email/call and schedule a private showing during the week — these worked out the best for me.

Good luck!

Enhanced by Zemanta
{ 1 comment }

Left the US for Oz

Library of Swinburne University

Image by WilLiao via Flickr

Do you ever get the feeling that your presence holds everything together, and that if you leave it’ll all collapse? This feeling almost never reflects reality, and if you get it frequently and think it’s true, it’s probably often sign of a narcissistic personality disorder.

That being said, I left the US on Monday, and in the subsequent 5 days the entire country seems to have done nothing but get itself into trouble. I’m not saying that if I stayed in the US that the debt limit crisis would have played out differently, or that the stock markets wouldn’t have fallen almost 5% in a day, or that the S&P wouldn’t have downgraded the US government credit rating for the first time in history… but the timing is interesting, isn’t it?

I’m currently in Hawthorn, which is a suburb very close to Melbourne, Australia, to begin my postdoc position at Swinburne University of Technology. Leaving the 100+ degree weather of Texas for the mid 50’s and rainy (today at least) Melbourne winter has been quite a jolt, but I think I’m really going to like living here.

I’ll provide more updates in the near future as I find housing and get settled!

Enhanced by Zemanta
{ 1 comment }

The Future of Education

I think that the future of education is an amazingly interesting idea to kick around. To begin with, I fully recommend watching the video of Salman Khan at TED where he talks about his Khan academy (I’ve embedded the talk below). This Khan academy is a nonprofit that he started after several tutorial videos that he made for his cousins on youtube started garnering an intense interest and following. He currently has over 2,000 videos that he has uploaded that teach a whole range of topics: from mathematics to biology to history.

He also has hired people to help him develop software that help to test students on what they are supposed to be learning. For example, to get through a module on simple addition, you have to answer ten questions right in a row — and you can ask for hints, or jump back to see a video on the concept you are having difficulty with. Also, once you get that done, you can go on from there to another subject.

The data is already impressive; in a few years, there are about one million students participating in the Khan academy and traditional teachers using the lectures to supplement the classroom. In most cases, the videos do the lectures, and the teacher looks at the dashboard for the class and figures out who is having trouble with what concept, and where. Further, she can see that another student has mastered the subject, and can use other students to help teach each other about the subject.

I’ve written about my thoughts on the future of education before, and I didn’t have a clear concept of exactly what the teachers would be doing if there was an adaptive computer model that instructed students based on their history and current understanding. Khan argues that this humanizes the classroom experience; I am surprised but I think he is correct.

I think Khan is missing one piece (or at least, I haven’t heard that it has the following piece): I think he needs to open up the possibility for videos from people besides himself. Now, this doesn’t mean that he has to accept any, and hell, I think he’s earned the title of Benevolent Dictator for Life on what is displayed by the Khan Academy. But there are a number of subjects that can be pressed into and teachers who are willing to contribute in useful ways that simply cannot be implemented by Khan himself.

Let’s just stick with physics, since I know physics. Let’s say that I thought I could explain the difference between electrical potential and electrical potential energy in a different/better way that Khan — so I create my own youtube video and submit it to the site. Does my 10 minute video module teach it better than Khan’s? I don’t know, perhaps yes, perhaps no. But the nice thing about having a million students, is that random testing can be easily implemented to display one video or the other to different students and to see what results come from it. Perhaps there’s not one video explanation that’s better than the other — just that people who are more auditory learners learn better from one video, while the opposite happens for the other students. There doesn’t have to be only one “right” video.

If people were allowed to submit videos, they’d clearly have to pass a minimum quality level, but after that level is reached, it would be interesting to be able to test and eventually implement the best video instruction over time.

In fact, I’m sure they have statistics on the videos that would make it easy to determine which video segments taught the concept they were aiming to the best, and which video segments students had the most trouble with. The Khan academy could have a “help us with your version of these 5 videos” that get put up for a week at a time, and as the new videos get cycled through, the bottom of the feeder gets filled with the “most need for improvement” videos. I think replacing from the bottom would be the simplest and most effective way for this kind of a thing to be implemented.

Finally, it would be awesome to have the courses like this be involved through the graduate level in all subjects that can be taught in this way — and I simply don’t think there’s time for Khan to be able to learn and regurgitate all that there is to learn in physics/math/history/biology/everything else. I understand that his goal might be to fully develop K-12, but there’s no reason to limit the benefits of the video database infrastructure to those levels and subjects.

Enhanced by Zemanta
{ 1 comment }

Moving to Melbourne, Australia

Melbourne's CBD in 2005 from Docklands at twil...

Image via Wikipedia

I accepted an offer to work at Swinburne University of Technology in Melbourne, Australia! I’ll start this August, and I’ll be there for at least 2 (probably 3) years. I’m working as a “Postdoctoral Research Associate in Extragalactic Fundamental Physics” — that’s the official title — and I’m very excited!

All I have to do is complete and defend my thesis by June.

If you’ve ever wanted to visit Australia and just didn’t feel like you had enough reason to get over there, make it happen in the next couple of years!

Enhanced by Zemanta

Using Dropbox and Git to write LaTeX papers

First things first — Dropbox is awesome. (Also, if you signup through that link, we both get extra free space… so do it). It’s a service that syncs a folder on your computer with a central server — and any other computers you have Dropbox installed on. But it does much more than that — you can share subfolders with other Dropbox users. For example, if I want to give my roommate a directory full of PDFs, I can just drag the directory into our shared folder, and Dropbox will automatically sync it so that he gets all the files without doing anything. Also, the files are on your computer — so if you lose internet or are on an airplane, you interact w/ the files just like any others on your computer.

Writing anything with LaTeX can be a trying experience. Writing an academic paper with multiple authors in LaTeX can be especially tricky. Dropbox comes in handy — but so does being able to use version control to track who changed what and when. Since Dropbox syncs files (and keeps a record of how the file looked in the past) it can sorta do the version control thing on its own.

Let’s just say you have the desire to use git (the fast version control system).

EDIT 1: Hannes (comment below) has warned that this is probably a bad way of doing these things.

Instead of Dropbox, use something like github to sync git repositories.

One trick I’ve found handy is to write LaTeX papers with each sentence on a new line — rather than putting returns on some number of characters. This way, diff and version control finds changes between sentences instead of just lines, which makes everything much easier to understand.

EDIT 2: Lelio (comment below) has suggested that this trick is unnecessary, instead use: git diff –word-diff. I haven’t tried it yet, but it seems reasonable enough to mention in the post itself.

I wanted for git to ignore a number of the helper files that get created when compiling LaTeX and found a useful repository of gitignore flags here. (Read the gitignore manual page here).

Easy enough to setup — I put it in my home directory, so keep that in mind if you put it somewhere else — so I did the following:

git clone git://github.com/github/gitignore.git

Now, I was actually confused about what to do at this point. After playing around with it for a while, I ended up making a bash script that keeps the git-ignore repository updated and referenced for my laptop. You can find the bash script here.

bash ~/Dropbox/scripts/git/gitignore.bash

Assuming you have my bash script in that location, that should create the .gitignore in your home directory. Next, you have to tell git that you want it to use the .gitignore file that the bash script created at ~/.gitignore for everything.

git config --global core.excludesfile ~/.gitignore

Here’s how I have cobbled together how I work with

On my laptop:

cd ~/Dropbox/Writing/
mkdir New-Project.git
cd New-Project.git
git --bare init
cd Project/location
git init
git add .
git status
git commit -m "First commit"
git remote add dropbox ~/Dropbox/Papers/2011-02-17-Whitmore-KeckCalibration.git
git push dropbox master

On another computer with a Dropbox account:

mkdir Papers; cd Papers/
git clone -o dropbox ~/Dropbox/Papers/2011-02-17-Whitmore-KeckCalibration.git

change some file

git commit -a 
  "First other computer edit"
git push dropbox master

Back on my laptop:

git pull dropbox master

work work work

git push dropbox master

Back on other computer the destination isn’t required:

git pull

work work work

git push

That should do it. If you have a better way of accomplishing the same thing, please let me know, I’m always on the lookout for doing things the right way.