Strangely, USB prices do not follow this trend. As of this writing, all USB units are the same price per gig (with variation) except one USB stick, at the highest end, which has the lowest price per gig of them all. But it’s on sale, so that may not count.

In the mean time, I’ve been wondering how much of my digital life I can get on a thumb drive. It turns out, all of it on one thumb drive, if I don’t count the archive of NEF photo files and music. The NEF photo files would need one more drive, the music half of another. If, that is, all three drives were 32 gigabytes in size.

Now, stop and think about that for a while. How many Voyager II’s would be required to hold that much data? How many cubic kilometers of old fashioned “drum” storage would equal the largest currently available thumb drive? Well, I went to look up some stuff so I could come up with a fun fact for this blog post, and came across a couple of … interesting facts.

The first fact is that the Internet is borked as far as actually knowing anything. Well, the Internet knows a lot of stuff, but it has a lot wrong as well. In this case, consider the following statement from some web site on the history of mass storage: 1963: IBM introduced the first storage unit with removable disks (IBM 1311), effectively ending the era of the punched-cards.

From this we know that after 1963, punch cards would be very rarely used. Perhaps in 1964 a lot of people still used them, then in 1965 fewer, 1966 even fewer, and so on. Certainly, if the “era of punch-cards” effectively “ended” in 1964, then by 1974, a decade later when I spent weeks and weeks of time with a large crew of people putting all of our data on punch cards to use it in one of the state of the art computing facilities in North America, we must have gone through some kind of time machine, because, well, there couldn’t have been any punch cards at that point in time.

Silly Internet.

Another site tells us that the first hard drive was the IBM RAMAC 305. But it wasn’t. The RAMAC 305 was a computer that used the first hard disk drive, which was the IBM 350. (Again, silly Internet.)

The first hard drive, the IBM 350 associated with the IBM 305 RAMAC computer, held five million bytes and took up 1.934X10^-9 cubic kilometers of space. So, it would take six thousand four hundred of these babies to equal the storage in one USB stick, if I got the largest USB stick available at amazon.com. Putting it another way, 50 megs of storage used to take up about 66 cubic feet. By those standards, 32 gigs would take up about 42,500 cubic feet. That’s ten semi-trucks.

I could have the math wrong, but you are free to check. Oh, by the way, the old data storage standard was 8 bits. We’re going to need some more trucks…

Anyway, it turns out that the current price of a USB is about two bucks a gig, based on several data points I cribbed from Amazon.com:

So a 4 gig stick costs about 8 bucks. The prices for the lower end sticks varies a lot because many of them have gimmicks, like they can survive being washed in the laundry, or they look like a penguin, or whatever. So the apparent maximum size as of this writing is 32 gigs, and that costs about 70 bucks.

However, I did see one stick on sale for about 32 bucks for 32 gigs. Is this the future? Will I be able to implement my plan to get everything on three sticks for less than 100 dollars?

The stick was on sale from a vendor I’ve never heard of, and it is the only one in the world (or so it seems) so I’m not going to buy that one. I’m sticking with the herd for now. I figure this signals a price shift happening at the top end. I expect/hope all USB storage media to shift to one dollar a gig sometime over the next (your guess is as good as mine).

In the old days, this was somewhat different. My data were on punch cards backed up on magnetic tape. I didn’t know exactly where the tape was, but I could look through the big glass window and see the tape library, and if I requested a set of data files, someone would mount the appropriate tape reel onto a big tape drive. When writing ‘software’ (including SPSS instruction sets, etc.) I could throw data and software into “scratch” which was a space on a hard drive (not the same as today’s hard drives) or a drum (a drum-shaped hard drive about the size of a Smart Car). The “scratch” space would be deleted on a regular basis, so I would put stuff there specifically knowing that I would not have to clean it up later. It would just go away.

Instruction sets/software and data could be stored on cards, magnetic tape, unruly disks or drums, or even paper punch tape (the mid 20th century version of the USB stick). But no matter what, not only did I (and everyone else) pretty much know where the data were, we also made specific decisions as to where to put the data to accommodate the not very automated process of accessing it or backing it up.

When I got my first real desktop PC (not counting my TRS-80) I ordered it (from my brother, who had access to such things) with two 20 meg hard drives, in an explicit effort to control the “where” part of this process. I configured the computer like this:

Drive C:
Main DOS system and software files
Scratch space (temporary folders, files to delete at next opportunity)

Drive D:
Files to use and now and then back up

This way, I could back up an entire drive and thus back up my data without all those pesky program files. The system and software, in turn, was “backed up” by virtue of me having floppy disks to reinstall everything. In those days, re-installation was not a lot more time consuming than restoring from backup, and allowed for re-configuring software and getting rid of crap software that one regretted installing to begin with.

You may or may not think that this approach was wise, but that is not my point. My point is that I thought about how to manage my digital life with explicit reference to where my stuff was, in particular, my data files.

Most people do this to some extent, but as shown in the examples above, not really. A lot of people probably think that the CD’s they copied into their computer are “in iTunes” as though iTunes was at thing that stuff could be “in.” Which it is not. Those music files are regular computer files no different at the system level than your c.v. stored in a file called “resume_version22_1982.doc” or whatever. But, they, the songs, are located collectively in a set of folders that are in turn organized in other folders in some arrangement that makes it impossible for a normal human to figure out, using folder and files with non human-meaningful names. Same with your email, most likely.

So no, thinking “My music is in iTunes and my email is in Thunderbird” does not allow you to find, copy, delete, read, edit, or otherwise access any of your songs or emails other than by using said software, and then, you can only do with those files what that software allows you to do. Which might or might not include exporting it for use in another format, with exporting capabilities determined by things other than what is technologically possible. Which, in turn, is like having a car that occasionally refuses to turn left for marketing reasons.

The degree to which this disconnect between you and your files is true depends on the operating system you use, and the software you use. Linux/Unix is all about directory systems, and is the most straight forward. In theory, ever single file that is yours is in one directory called “home” or a subdirectory thereof. This is in contrast to Windows, in which your files are … somewhere else, and the system seems to be designed to make it as difficult as possible for you to find them.

This does not mean that you can easily find everything that you might consider a “file” or similar entity in Linux. There is a good chance that your email software uses some bizarro file that you can’t easily see inside of. (I use alpine which puts the emails inside a text file, but hardy anybody does that.) There are “hidden” files in Linux just like in Windows (in Linux, everything that starts with a “dot” (“.”) is automatically “hidden” …. meaning you can’t see it unless you “unhide” that which is hidden). There are other strangeosities as well.

This where to put stuff issue came up when I had a conversation with my brother the other day about backing up. We probably have different backup philosophies. My brother is an actual computer expert who runs other people’s computer and so on, so to him backing up means doing what you need to do so that you can restore the state of the system … software, settings, and data … accurately and quickly when something breaks. To me, backing up is securing the data files (including pictures, documents, whatever) and I do not really care about the software. Of course, there is an overlap. The programs and scripts I’ve written are data, not software, by this thinking.

Also, I have at least to kinds of data: Stuff that does not change and stuff that could change The stuff that does not change includes photographs downloaded off a camera. Yes, I can modify those but I’d generally leave the original intact. Also, all the documents, such as presentations and handouts, for a given class I’ve taught are archived, for various reasons. the original is left untouched and a subsequent year’s material modified from the originals (in theory).

So, there are two kinds of data: Archive and dynamic. In backing up, I can add to the archive without having to check if anything previously backed up has changed, because it hasn’t. But to do this, I have to have a space where I put archive material… a separate directory or device where such things go. That might seem dumb, like extra work, like I should have the computer just back everything up or otherwise automate the system. But consider this: Not counting .iso files (which are a whole ‘nuther issue) I have about 194 gigabytes of “archive” material and 10 gigabytes of “dynamic” material. Is it really smart for me to run a ca 200 gigabyte backup every day, or even every month? No, of course not. Better to have the dynamic stuff backed up all the time, and to maintain the archive in a much slower, more ponderous, but still effective manner. If I’m using an online backup, I don’t need to verify that 200 gigabytes of data is the same as it was yesterday, or last week, or whatever if I know it is. Also, any change in archive is not routine … it is something broken. Once something is in the archive, the copying is always one way, out of the archive from a given source. My archive is worot (write once, read only thereafter).

So this all leads to two pieces of advice: 1) Make choices that allow you maximum direct access to your files (like using Linux instead of Windows, or appropriate choices of application software) and 2) Divide your files and stuff into categories that have to do with the nature of the backup and your access to it, and then by topic. For me, my desktop is my scratch space, Dropbox is my dynamic file space, and I’m not going to tell you where I keep my archive. This means that I have photographs in two places: The archives (downloaded off a camera, left alone) and the various cropped or otherwise revised, and in some cases simply selected and resampled or compressed photos. I have course material in two places: Archived away but accessible, and dynamically changing until I’m done with it (then it goes in the archive).

How do you store your files? How do yo do backups?

ADDED: It has been pointed out (below, in comments) that iTunes now organizes your files in a sensible human readable way. Good for iTunes I was working with older information.

The same commenter suggests that email must also be organized that way. Don’t believe it. Especially if you use Outlook!!

In any case, it is still true that your files are where the secondary software puts them, not where you want them, necessarily. Which is not an entirely bad thing.