Thursday, September 3, 2009

Year 5 Day 3 - File Management

I rebuilt my PC a couple months ago - my old one was a few years old and had some hard drive issues. Luckily I had backups of all my data -actually it wasn't luck, I make sure to have multiple copies of everything.

Read on for more.

I won't go in to too much detail but for the sake of this post - when I get home from shooting somewhere if the day was awesome, I off load my cards to my pc, rename files, and make a backup copy of the files, and then move the files to current location for originals.

I try to make sure my 2nd copy of the images is either DVDs or now, a second hard drive which is NOT in the same PC as the originals. I use a 1TB NAS which is full, and another 2 x 1 TB external USB drives.

So, lately I've been juggling around my files and trying to organize them better. I think I will use some of this month off from flickr to redo my scheme from scratch.

I started to use Lightroom to do all my file management a few months ago. Picasa would just consistently crash and erase the entire database of images and have to rescan everything. I know it was because I had too many files, well over 200,000 images / originals, and maybe over 300,000 files since I had shot jpgs + raws very often. And I have each high level folder as its own Lightroom catalog to keep the catalogs somewhat manageable.

The problem is that stuff is now all over the place, the catalogs are roughly:
1) 2005 through part of 2008.
2) Part of 2008 through part of 2009.
3) Most of 2009.

And for the 3rd grouping I began to store the folders from each day in a parent folder for the location. Which is good for the new stuff, but now my old stuff that is just sorted by day isn't part of this setup.

One of the challenges I face now is to reorganize the first two catalogs in to groups by location and then day. Many many of those day-folders are for multiple locations-days when I went to two or three spots. And lots of the folders are named things like "2006_07xx_MDMisc" which might be multiple days and a variety of stuff. I guess I can sort through it all, but THEN the matching backup copies are going to be differently named folders.

I could redo the backups, by deleting the backups and re-copying the freshly organized folders but if my main copy (the hard drive) dies during that process I might lose ALL the copies.

I might just have to buy another couple 1 or 2 TB hard drives so I can safely juggle EVERYTHING around.

One of the roles I have at my day job is data management and protection strategies and technologies, and testing and vetting it from end to end. The last thing I want to EVER happen is to lose some of MY DATA! So that is why this might all seem a little overkill.


And then I need to determine the impact of moving files around from one catalog to another , if those folders had lots of editing applied to them in LR. Most of that is just recent stuff from the past few months, so it might not be that bad. I could lose all the editing work and not be too upset. I always make my edits, and then export a full resolution jpg from Lightroom.


If I do go through all this hassle to move files, rename folders, regroup folders I want to do it right. A couple of the possible plans I have right now are:
1) Sort everything by year, and be done - folders will then be for a location and day (never 2 locations per folder). Keep a LR catalog per year.
2) Sort everything by location, and for places like the DC Zoo, or Great Falls, where I've been 50 or 100 or 200 times I'd make that get its own LR catalog.

And something that just now occurred to me, I need to develop a plan for storing my HD video files.

All that, and I plan to swap out my temporary install of Windows XP for Windows 7 in a month or so. That's going to rock - my AMD 3Ghz 4-core CPU with 8GB or RAM and 5+ TB of disk! Woot.

Ok, I've got my work cut out for me, and I won't stand for losing any data...


That's enough to make your head spin huh?
Summer Blur 2009



MacMoov said...

Jon, Since you are rebooting your storage strategy, you may want to read the Peter Krough DAM book first to give you some ideas on how to organize your media.

It really helped to build my file management methods.

Nikographer said...

Thanks Mac, I probably *should* look to someone with more experience than me for help ;)

AndrewN said...

Jon, Here's a strategy that I picked up from another professional. Keep LR catalogs per quarter, grade every photo in the catalog as you shoot.

At the end of a quarter:
1) backup the quarter to an external copy - DVD.

2) export all 4/5 flagged photos into a separate catalog: your good photos from that quarter.

3) Create a set of CD/DVDs containing jpegs of all the photos in there and file the CDs+a printed index of the files with the copyright office (proof of ownership and you can now collect damages should the photos get stolen by a magazine etc).

4) Import the quarters good photo catalog into a full catalog of all your good photos that you can search by date, keyword etc. You can still go to the quarter if you need all the photos you took that day.

5) Remove the last quarter LR catalog from your HD.

It's not that difficult to do and the big catalog only has photos worth doing further processing work on.

Keep up with the great blog posts, I enjoy them and your work, cheers.

Nikographer said...

Ok, now I just need to figure out the mechanics of steps 2 and 4.

I've shuffled my folders around, and have them grouped by year, and a catalog per year.

I really do like the idea of having multiple catalogs, and then promoting the best images to a higher level catalog.

It's going to be tough to catch up on the ratings, and keeping at it for new shoots. But I suppose that's the only way. The sorting won't magically take care of itself. And over time my problems will just grow to a completely un-fixable state.

Thanks for the tips Andrew. I appreciate it.

AndrewN said...

Create a "Smart Collection" between dates with flag >= 4. Use the menu popup from the collection to "Export this collection to a new catalog". Make sure that all the metadata and keywords are included in the exported catalog. Cheers.