CONSOLIDATE

The first step in organizing a new repository is getting the files all in one place. The goal is to clean off all of the various drives on which they are found. In this step, duplicate files and directories will be created.

Do not worry. Eventually, duplicates will be removed. Until then, err on the side of caution so, in your zeal to clean and purge, nothing important is lost.

Assemble the Ingredients

Before starting, collect the following items to be used in the process.

  1. All field drives (e.g. mobile hard drives, USB drives and CF/SD cards)
  2. Any cables (USB3, Thunderbolt, FW800, FW400), power supplies, docks or card readers required to connect your field drives to a workstation.
  3. All Aperture/iPhoto libraries stored on all of your workstations and laptops.
  4. Three large cardboard boxes.
  5. A permanent marker.
  6. A workstation with all of the ports (Thunderbolt, Firewire, USB3) required to access your various field drives.
  7. Two, clean, clear DAM Drives.
  8. Power drill and metal drill bit.

Dedicate A Workstation

If you have an older, seldom used computer, consider making it your dedicated consolidation machine. As previously noted, this process can be time intensive; using your primary workstation increases the chances that the process will be interrupted. Ideally, this system supports Thunderbolt and/or USB3, which make the process go faster.

If you must use your primary workstation, execute the Data Consolidation step at the end of the workday and let it run overnight. This takes longer, but reduces the chances the process is interrupted by trying to multi-task during the workday.

Build or Buy Two New DAM Drives

I built two 12TB Drobo 5Ds that, with Drobo’s built-in data protection, gave me 8TB of storage for each unit. Pick up the biggest file stores you can afford. One can never have too much drive space.

If you have existing RAIDs, you can get away with a folder on those systems if you have the drive space. However, a new, dedicated DAM drive reduces confusion. If your current archive is small enough, you may be able to get away with something like Seagate’s 5TB Expansion Drives. If your system supports Thunderbolt or USB3, make sure your new DAM drives support these technologies as well.

The Physical Workstation

Remember those three large cardboard boxes? With a permanent marker, mark the boxes as STRONG, WIPE and CLEAR. Place SOURCE to the left of your work station. Place WIPE immediately to the right of the workstation. Place CLEAR immediately to the right of WIPE.

Create Your Repositories

In the root directory of each DAM drive, create the following base repositories (folders):

  • 00_LIGHTROOM (Lightroom Catalogs)
  • 01_PROJECTS (Working Media Files)
  • 99_SOURCE_ARCHIVE (Source Media Files)

The two number code at the beginning of the repository name serves two purposes. First, it causes the repositories to line up in the file system in order, with the most frequently used (00_LIGHTROOM) at the top of the directory. Second, this code structure will be used to help group similar projects in subfolders.

When rebuilding our DAM, the 99_SOURCE_ARCHIVE repository is used as catchall for media currently spread across our workstations and field drives. When finished, it will act as our “deep” archive and 01_PROJECTS will house the final, working repository.

With base repositories in place, create subfolders that suit your post workflow. When finished, your source and DAM drives will look like this:

My repositories center on specific projects grouped into broad categories like 01_FAMILY and 02_CLIENTS. Notice that media is not sorted by type (e.g. photos, video, audio) at this level. That is done at the project level as shown here:

There are many ways to arrange your folders. Find one that makes sense for your workflow.

Assemble Source Drives & Libraries

If your existing Aperture/iPhoto libraries are already well organized, move them using Rich Harrington’s “The Essential Guide to Moving an Aperture Library to Adobe Lightroom“.

If your Aperture and iPhoto libraries are a mess, move them to a dedicated folder on your workstation. If these libraries are Managed, it is as simple as copying the library into a new folder. If the library is Referenced, consider converting it to Managed first. This puts everything in one place and ensures nothing gets lost in the move.

If your consolidation workstation is not your primary workstation, copy these Aperture/iPhoto libraries to one of your source drives.
Place all of the source drives collected in Step #1 in the SOURCE box.

As you fill your SOURCE box, don’t rule anything out. The goal is to get it all in one place.

Data Consolidation

Connect a single source drive to your workstation. On the source drive, create a new folder called 00_MOVED.

Resist the temptation to connect multiple source drives and start multiple copy processes. First, pushing to/from multiple drives only slows the copies down as it fills up the workstation’s data “pipe”. Second, by taxing the system, you increase the risk of data corruption. Third, multiple source drives make it easier for you to lose track of what has been copied and what has not, which increases the chances you will create duplicates you will need to clean out later.

One source folder at a time, start copying data from your source drive to the appropriate folder on BOTH of your new DAM drives, which creates mirrored volumes for safety. To do this, I used Red Giant’s Offload. I love this tool for two reasons. First, it automates backups to two separate DAM drives. Second, as it makes the copy, it does a checksum on each file. This ensures a clean, reliable copy. It takes a bit longer, but the peace of mind is worth it.

When the DAM drive copy is complete, move the copied folder into the 00_MOVED folder on the source drive. This helps you keep track of moved folders as you process the drive.

Repeat this process with all of the folders on the source drive until everything is on both DAM drives and in the 00_MOVED folder on the source drive. When finished, your source and DAM drives will now look like this:

When the entire source drive has been processed, disconnect it and place it in the WIPE box. DO NOT erase the drives yet. You still need them as backups, albeit messy ones, for a little bit longer.

Rinse & Repeat

Repeat this process with each drive or card in the SOURCE box until it is empty.

NOTE: As you process your source drives, you may find what appear to be duplicate folders & files. When this occurs, do not copy over or merge the folders on your new DAM drives. Doing so makes it very likely that you will corrupt or erase data.

Instead, append a numeric sequence to the duplicate folder (e.g. 2012_BAHAMAS, 2012_BAHAMAS_02, 2012_BAHAMAS_03) then copy the apparent duplicate as well.

This is the one time we will knowingly create duplicates, but we are doing this for safety purposes. We’ll cleanly purge duplicates in a later step.

When you are done, you’ll have an empty SOURCE box, a full WIPE box (acting as a temporary archive) and roughly organized backups of everything on the DAM drives. It will take a while to complete, but you are now ready to start cleaning things up using the CONFORM and CULL steps.

Doug DaultonDoug Daulton is a writer, photographer and filmmaker based in Las Vegas. His current project is Bokeh, an independent feature film currently in post-production.

Follow Doug’s work: articles · web · twitter · google+.


This Post Sponsored by:

Drobo. A family of Safe, Simple, and Expandable storage systems for capture in the field, editing in the studio, or backup and archive.

lynda.com Learn photography anytime, anywhere, and at your own pace—from bite-sized tutorials to comprehensive courses. Try lynda.com free for 10 days by visiting lynda.com/Photofocus.

SongFreedom is about artists supporting artists. We’re a music licensing platform with the best music available–stuff from the radio, or your favorite indie bands and soundtracks.  A place where photographers and cinematographers can find the most powerful song for their story with the click of a button.

Capture Cinematic Weddings with Ray Roman.  A Wedding Workshop Series Traveling the US and Canada. Learn to shoot stunning wedding videos from the master of ceremonies, Ray Roman — The World Renowned Wedding Cinematographer! Use the code PHOTOFOCUS for discounted tickets.

The HDR Learning Center Check out new ways to use High Dynamic Range photography to make compelling images. Free tutorials and posts to get results. Produced in partnership with HDRsoft.

Join the conversation! 2 Comments

  1. Okay, I am a dummy. Why do I need a “Power drill and metal drill bit”. Perhaps, this was a joke line

    Reply

Let us know your thoughts...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

About Doug Daulton

Storyteller; words, stills & motion. Bokeh, a sci-fi feature shot in Iceland, is Doug's current project and is now finishing post-production. When not telling stories of the fantastic, Doug loves putting the natural world – from waterfalls to wild horses – in front of his camera while traveling the world. For regular updates, follow Doug on Twitter, Instagram, facebook, Google+, tumblr or his blog.

Category

Adobe, Gear, Photography, Software, Technique & Tutorials

Tags

, , , , , , , ,