Organising Photos
Overview
Dave Bradford has been on it again but this time on file or more specifically photo management. My existing library contains camera shots from film days scanned in to digital shots from things like phones totally 44GB. Organisation of my files have been pretty lax and has resulted in various file structures. The idea is to move them to structure which can be managed using just a basic file browser to any sophisticate program like Lightroom or Picasa.
Concept
The idea is to use a combination of files and folders to orgranise the pictures in such a way that it’s fairly easy to find and potentially automate photo orgranisation in the future. This is predominately aimed at photos but I can’t see why it can’t be extended to documents, moves/films, etc.
The future structure will be organised in \year\month\year-month-day hour.minute.seconds_copy.extention. The year and month allows a general organisation by the date. The files will have the full date time stamp on them so should they be moved by accident it can be moved back to the correct place. Also note the dates use hyphens to separate the date values and the time uses dots. This is so the files can be searched on dates only, time only or combination of both based on the separator. The underscore also separates the copy number. For example in a burst shot photos my be taken within the same second so more than one file will have the same name. The copy number gets around that but also the underscore allows for searches to be done where there are more than one photo of the same second. Handy if you want to later prune similar shots.
The logic behind creating folders by year and month is so that there are not too many folders. Going to date level will create lots of little folders whilst sorting them by just year means there will be a lot of files in one folder. Using year and month seems like a good balance. Also the less folders there are the easier it is to index and perform directory scans say for backup purposes.
My Setup
I have a top level folder with all my photos in. These include scans from physically printed pictures to screenshots taken on a computer. Not all of them have EXIF data and also the date time stamps are not accurate because of various formats and re-installs over the years.
The files are all over the place. At first there were some in a folder on their own like pictures of Danny go in the Danny folder. Then it evolved into yearly folders. Later it then went by location with the advent of geo tagging. So a folder of the location like London and then a sub folder of the event or year or just lose in the London folder.
Preparation
Before starting it is essential to have a backup of your data. If not do not even attempt to do this!
I had a backup as well as working from a copy of my library so should anything go wrong i could take another copy of my local original. If something went really bad then I could restore from an external drive.
Tools
DropIt can batch organise files based on associations (rules). The program can have multiple profiles with different associations. It’s more suited to on the fly batch management rather than organising heaps on old files. I say this because it lacks some flow controls in it’s rules like do this, then do this, then do this. Rather it has a set of rules, it runs the actions if the rules match and then it stops. You’d have to run another profile with different set of rules should you wish to do multiple actions.
DropIt has an always ontop window which allows you to drag files or folders onto and then it will bring up a before and after. Based on this you can select the files to skip and run the associations. DropIt is Free and open source software.
IrfanView is an image editor but also has batch tools for managing images specifically. It can read IPTC and EXIF data and perform batch renames as well as batch conversions. It can also search for files based on name or meta data and the results can then be selected into one of the batch modes for renaming or convertions.
ExifTool GUI is a free tool which can batch edit meta data from EXIF to XMP. The tool comes in 2 parts. ExifTool is a command line program and the GUI which is a graphical interface that sits ontop. The interface part will not work without the command line.
Remove Empty Directory (RED) scans folders and sub folders for folders with nothing in them. It will show a list of results from a scan and has the ability to delete them.
File Types
I used IrFan Viewer to find all the files which are in bitmap format (bmp) so that I can convert them to Jpeg. Select all the files in the results which require conversion and press the Show in Thumbnails button. A grid of the pictures selected will appear whilst the search dialogue will disappear.
Select all the images, right click > Start batch dialog with selected files…
Unfortunately, Irfan won’t create the new files in the same place as the originals so they will have to be moved back in afterwards. Set the conversion settings such as the quality and output directory.
File Rename
Renaming the files to the correct date time should be the first step. This is mainly for files without EXIF information and using the last modified date. If you modify the meta data before doing this then the last modified date will also change.
I used DropIt to rename the necessary files. I created 2 profiles to get what I wanted. First was a profile to rename files based on EXIF data. The associations on the profile are set to scan .jpg and .png files for the meta data. It will delete any Thumbs.db (optional) and ignore any video files.
I then dragged directories onto the DropIt floating icon which will bring up the processing window. Expand the window to show the before and after by pressing the green arrow. The list will allow you to highlight, right click and select skip to not rename the marked ones.
Depending on how old or well preserved the photos are this will take a very long time because I could not find a way to not rename files without EXIF data.
The second profile had the same associations except it took the last modified date. I found the last modified date to be the next best option because the creation date may be when a backup copy was restored. To speed things you can copy associations from an existing profile to another.
To do this, switch to the profile with the association and go to the associations window. Right click on the association you want to copy and select Copy to… It will prompt which profile you wish to copy to.
Once complete, do the same as before but this time checking the last modified dates are as correct as they could be.
Meta Data
Go to Phil Harvey’s site and download the Windows Executable version. At the time 9.52 was the latest version and downloaded as a zip file. Unzip the download and rename the file extracted file from ExifToolGUI(-h).exe to ExifToolGUI.exe. Note down the file location as it will be used later.
Next download the GUI program from ExifTool GUI. Again unzip the file to a folder. In my case it’s called exiftoolgui. Copy ExifToolGUI.exe into the same directory as the gui program.
Start the program using ExifToolGUI.exe. The program isn’t the prettest thing but it works. It’s split into 3 columns. The left is a list of the location directory. Use this column to browse to the photos. The middle will list the files in the selected directory from the left column. The right column will show the meta data (if any).
At the top of the (right) meta data column are buttons which allow you see the different meta data format (EXIF, IPTC, XMP). Under these meta data format are buttons to edit them.
I turned off the file backup by default. With backup turned on any files being edited will be copied and renamed to .backup before changes are made using EXIFTool GUI. To turn it off go to Options > Don’t backup files when writing.
One of the main purposes I wanted to use this tool was to add GPS co-ordinates to the photos. Not all will be accurate but I have organised some of my photos by location and by year so I wanted to preserve as much as possible before I moved them. This task will take the longest because it’s manual and laborious.
To use Google maps change the default GPS from degrees, minutes to decimal degrees. This is optional but Google uses decimal degrees. Also this setting is not saved when you exit the program so remember to turn it back on.
To find the location you want the GPS information for by zooming in as much as possible. Right click on the point your chosen point and select What’s here.
The faster way is to use the new(2014) maps interface. At the time this was not rolled out to everyone but I found using Chrome to view Google Maps increased the chances that Google will serve the new style maps. Again zoom into the location but this time click on the spot. A card will pop up with the co-ordinates at the bottom. Be patient because the map may still be loading (with no indicators) when you click and nothing loads.
The search bar will appear with co-ordinates in degrees which need to be converted.
Due to the clunky design of Exiftool GUI the GPS co-oridinates are not a straight forward copy and paste. The decimal place and formatting often causes the wrong value to be entered. Using decimal degrees, if the latitude value (usually the first number) is a positive number then it’s above the equator. In this case the North tick box should be checked. If the longitude number is a negative number then the East tick box must be checked.
Use a converter like this Google page to convert the GPS co-ordinates if necessary.
Move Files
Using DropIt again, files can be moved into a year\month structure very easily. Create a new profile and add the following associations:
The first association uses regex to match any files in yyyy-mm-dd hh.mm.ss format and move it to a directory taking the year and month from the file name.
The last association sets it to ignore any other files for example ones that have not been renamed into date and time.
As most of the hard work was done in the file naming, the move is fairly robust and a bunch of files / folders can be done with relatively little effort required.
Once the files have been moved to their new structure hopefully many of the old directories will be left with no files in them. This is where the RED comes in.
Future
Once completing (I’m still in the process of doing it) I should be able to perform one click photo management for any new photos or videos I get. Using DropIt has allowed me to rename as and when necessary and move them to the correct destination.
An improvement I have already noticed is my Android phone uses the same naming convention as above so that’s one less step needed and using Dropbox to automatically upload the camera roll means it all appears on the PC ready to be moved into their respective folders. Also GPS tagging is a lot more consistent compared to my previous phones.
Summary
One remaining issue is how or where to put files which have been scanned or have no date or time information available. Currently I have moved them to a folder to be sorted but I cannot envisage when they will ever be done.
The next step is to look at other things which can benefit from something similar.
Pingback: Document Organisation Part 2 – Management | Danny Tsang