22 November 2008

The Case for a Logging Standard

Logging astronomical observations is a very personal thing, and not all observers keep a log. Perhaps visual observers are the least likely to record observations but many do. Most imagers at least keep records of their exposures, and the end product - an image - serves as excellent documentation. Sketchers probably keep the best records of all since their work is, by definition, recording detail manually. With observers' logging habits so varied, why do we want or need a standard?

From an observer's viewpoint, it may be less than apparent. Many of us record observations so that we can refer back to them and determine whether we've seen something before, or how it looked in the past, or when and where we saw it. Others collect observations and present them for an award. Still others want to publish particular observations on a website or blog. This really defines 3 different usages: searching, managing and reporting. There are, of course, many ways to meet these use cases: using pen and paper, a word processor, a spreadsheet or a database are the usual suspects. Any of these work well as long as we want to stay within the same process - a closed system in the algebraic sense.

So what if we want to venture forth from our tried and true logging practice? Perhaps we want to contribute an observation to a scientific collection (e.g., AAVSO), or publish them onto the web, or enter them into a program that offers a new, desirable capability? These cases require re-entry of the logged data to some extent, a tedious process at best. Worse yet, what if your log becomes inaccessible? Think of a hard drive crash, lost installation disc and a program that is no longer supported. Yuck.

Enter a standard file format for observing logs. The first three cases mentioned above are likely to be solved by having a standard because the more common the use case, the more likely that someone can, or will, solve the problem at least once. As a developer, I would be happy to support one exchange format, but supporting one for every file format requested by users is hard to justify.

The disaster scenario described above is perhaps the best reason for users to demand a standard log file format. I like to think of it as a warranty for the observation data I have spent years accumulating. If a program I have used to record observations either stops being updated or becomes less appealing, at least my data can move forward with me via a standard format file.

Developers will only support a standard if it is a marketable feature. That means that users will have to demand it as a must-have feature for a standard to gain the traction that makes one ubiquitous.

I wonder how many more observers would record their observations if they could enter them once and be assured that they wouldn't have to re-enter them time and time again. I also wonder how many scientifically valuable observations are lost because they can't get out into the astronomy community.

Until there is a standard available for users to demand and for developers to support, the arguments above are just conversation material. Fortunately there is a candidate under development. More on that next time.