BackupPC second install

Backuppc setup summary

Had to reinstall Ubuntu due to disk crash (!) on the backup server :-)

Installed 14.10, upgraded to 15.04
Removed unity then xorg to keep a text-only server.
Added openssh-server, backuppc, and their required packages.

Reset password for users admin & backuppc:

cd /etc/backuppc
sudo htpasswd htpasswd admin
sudo htpasswd htpasswd backuppc

Problem 1

Connecting to http://base.lan/backuppc says I want to download a bin file...
Actually Apache2 knows only the hostname base not base.lan
Added base.lan to /etc/hosts as alias for base: still only the BIN file for base?lan (but base/backuppc works :)
Might be a browser cache issue, in the meantime I can use the short hostname fine.

Problem 2

Copied localhost.pl from the previous installation, it doesn't work.  This is a tar backup and its getting permission denied on a number of directories/files under /etc
So added a permission via visudo as explained in the FAQ http://backuppc.sourceforge.net/faq/localhost.html

Problem 3

Now adding another (linux, ssh+rsync) <host>.pl file.  Also need to add it through the web interface otherwise it's not recognized. Then also need to create a subdir for the host under /var/lib/backuppc/pc/<host> to hold the log file etc... (should the web interface do this?)
After this the new host still doesn't backup (ping failed)
Turns out that the permissions for the directory /var/lib/backuppc/pc/<host> need to be 750 not the default 755.

Problem 4

Now the backup is starting and immediately failing for the new rsync host.  Which was expected since the ssh key stuff has not yet been done.  So followed instructions as per https://help.ubuntu.com/community/SSH/OpenSSH/Keys to generate a new key pair for backuppc and then scp the .pub to <host> and append it to authorized_keys in root's .ssh directory.  That seems to do the trick now the first full backup is running.
The first backup took a really long time (>24h) but this was my main data storage PC and I'm happy to have it backed up again :).  The next backup (incremental) took only 30 minutes which is fine.

Problem 5

Added a new (Windows+SMB) host.  Didn't work initially, the problem was in the both the config file (username, password, sharename, directory names) and on the client itself (match all of the above with the actual disk and OS configuration).  For example I had to reset the user's password on the OS side and put the new value in the config file.  Now the first backup is running.

More Darktable efforts

Installed DT 1.6 using the PPA for Ubuntu 14.10 : went perfectly fine.

Processed a few shots taken both with and without the Pentax K-r "Highlight Correction" and "Shadow Correction".

Supposedly the Shadow correction only affects JPEG (I was shooting RAW only).  But the Highlight Correction definitely underexposes the picture by 1 stop (by setting the base ISO to 200).  The JPEG preview is automatically compensated for this but Darktable doesn't seem to support and a +1 EV exposure compensation has to be applied during RAW development to get to the same kind of exposure as when this Highlight Correction stays off.

I then processed a RAW from the E-PM1 that had been severely overexposed during the shoot.  Applied about -1.0 exposure compensation and the highlight recovery modules in DT and the results were much better than what I could get from the JPEG (from the camera, this was a RAW+JPEG shot).  Especially when it comes to skin tones.

SO the conclusion would be: disable Highlight Correction if shooting RAW (saves manual compensation during development, and compensation can still be applied manually on the really badly overexposed ones with some kind of success even without this.

More RAW workflow options - Geeqie

In the previous article, I tested some RAW processing with a moderately complex workflow due to the camera (XZ-10) not being supported currently by Darktable so I had to convert to DNG etc...

I now have installed geeqie, supposedly it can view also RAW files and thus could be used for culling images even before importing/processing them.

So I empty the card to a folder outside Shotwell's monitored library folder and try geeqie.  And geez yes for that purpose it works very well.  I tried with the XZ-10 pictures already mentioned above and some new DNG from the Pentax K-r.  I had shot a few in rapid sequence (5 pic/sec) and Geeqie allowed me to zoom in on the subject, and *really* fast scrolling (also with the mouse wheel) to check which are the best ones.  The others can be deleted from Geeqie directly before any import.

Geeqie also can add keywords to the RAW files.  It has a hierarchical keyword structure and keywords (aka tags) can be added to one picture at a time or (using right-click) to all selected pictures from the current folder.

Keywords and other metadata are stored in .xmp files (can be switched on in the Preferences).  Geekie keeps track of these files and pairs them in the GUI with the appropriate RAW file name.  Same for JPG files if any (not in my case today as I was shooting RAW only).  Pretty neat.

Other metadata that can be set from Geeqie include a title and a description.  It is also possible to set these fields on a single or on all selected images (right click) or add to existing field values instead of replacing them.

Importing into Shotwell: none of the metadata show up :-\ because probably shotwell doesn't even try to read the .xmp file.  Shotwell created the usual _DNG_embedded.jpg files.  They're about 1.6 Mb in size (larger than those from the XZ-10 thus for the same resolution, still not very big but I couldn't really get bigger ones out of the box with Darktable).

Opening with Darktable, importing the DNG files from the folder, tweaking a bit then exporting with or without a suffix: Shotwell imports (after a restart) and sees the metadata from Darktable.  So far so good.  Keeping 1 star for the DNG and setting 2 stars for the JPG : so I can work on tweaking metadata a little bit more if needed before upload.

Cleaning up: rejecting and deleting a few bad ones in Shotwell deletes the DNG_embedded jpg and the DNG files, but also creates an empty DNG_shotwell.jpg (!) and doesn't remove the .xmp files.  And Darktable still needs a separate cleanup step (which will remove the .xmp files then).

Conclusion: the JPG files are not paired with the RAW in Shotwell, they contain metadata and are developed by the RAW processor.  However the whole workflow is still pretty much a chore.  And each image appears twice in Shotwell of course.

RAW workflow options - more testing with Shotwell RAW support (this was nasty) and using Rawtherapee

Shot RAW+JPG (mostly to be able to import separately in order to compare camera output to what I get with different kinds of RAW developments). I renamed the *.JPG to *C.JPG (for Camera JPEG) in order for Shotwell to see these as independent pictures and not link them to the RAW files.  These were ORF from the XZ-10 so I know Darktable will need a DNG conversion to use them.

Imported 6 files with Shotwell.  They got in the library, Shotwell extracted the 2400x3600 or roughly 8 mega-pixels preview images.  They are visually almost the same as the 12 mega-pixels camera JPEGs even full screen (this is not a big surprise as my screen is 1680x1050 or only 1.7 mega-pixels.  However there is more fine details in the camera JPEGs so I bet the compression/quality is compromised also in the preview JPEGs.  They are also only about 5 times smaller than the camera JPEGs!  There's also quite a few differences in the EXIF data from these embedded JPEGs.  Not that interesting to me though.

I then reconfigured Shotwell to import in folders named %Y/%Y-%m-%d (2015/2015-01-01 for example) which is more compatible with Darktable as this uses the folder name as "roll" id.

I renamed the "09" folder into 2015-01-01 and moved it under ../2015/ where it should have been.  My 6 images went to the "missing" folder in Shotwell, along with the 3 "ORF_embedded.jpg" that got extracted from the RAWs.  Navigating Shotwell to the ../2015/ folder, it was failing to notice the new 2015-01-09 folder under that.  No apparent function to ask for a refresh so I stopped and restarted Shotwell to get at my images again.

After restart, Shotwell found my 6 images back but decided to re-extract the embedded previews for showing me the RAW files.  These ended up in *_ORF_embedded_1.jpg files.  But I also got new _ORF_shotwell.jpg files which are bigger than the *ORF_embedded.jpg files.  About double the size and a resolution of 11.6 megapixels.  It also found and recataloged in a separate event only 2 of the original *_ORF_embedded.jpg (why?).  I removed all 6 of these embedded jpegs and shotwell happily recreated them (without the _1 suffix though. And the 2 extraneous *ORF_embedded.jpg files went to the "missing" files where I removed them from the library.  So far so good, I have my 6 images in Shotwell in the event for 9 january.  But I also have an extra 3 jpegs. 
Plus, when I tried to view the ORF files I got a message that the file was missing, and they went all 3 to the missing files as well.  I was left with the camera JPEGs only (*C.JPG).

So I moved the ORF files outside shotwell's monitored tree and cleaned up the remaining extra .jpg files then cleaned the remainings from the "missing" section, also found and removed some in the trashcan section, and reimported the 3 RAWs.  So far so good, the _embedded.jpg files came back and strangely enough the names of these files now appear in Shotwell under the image (instead of the .ORF suffix).  Then I noticed that my camera JPEGs were gone... Did I deleted these manually (I think so).  Go figure... so got them again from the memory card and started over again.  Cleanup, import from card...  Again I got two separate events (one with the embedded.jpg and one with the rest).  Trying to open a ORF file ended up in a shotwell crash...

Restarted shotwell with logging activated. Tried re-import, no joy (again 2 events).  Cleaned up and renamed the RAW files on the card with R.ORF suffix and re-imported (shotwell failed to import the OLD names, apparently it hadn't noticed the file system level rename that I did just before).  Cleaned up again the restarted shotwell.  Re-imported (now it was seeing the new names with R.ORF suffix).  This time it went fine, except that one of the _embedded.jpg was again in a separate event... not 2, not 3. I removed it from the library (no delete). Then merged the remaining 2 events into one.

Ok now it looks fine.  The *C.JPG displayed by Shotwell is slightly sharper than the equivalent *R.ORF as expected.  The embedded.jpg files are there (only 1 per RAW file) and are still 20% of the size of the *C.JPG.


Changed the developer from Camera to Shotwell.  Little pause, the _embedded.jpg file is removed and a _shotwell.jpg file appears.  It is color (not B/W like the camera JPEG) and clearly lacks lens distortion correction and contrast/color punch.  It is significantly larger than the old embedded.jpg file and the resolution is a little larger than the camera JPEG (probably due to the lack of lens distortion correction which implies a small crop).  I adjusted contrast and removed color (saturation->0) and compared.  Appart from the lens distortion this is not so bad but despite the resolution increase the camera JPEG is still sharper (and over double the size).  Not very good thus.

So the next test was to set my RAW editor preference to Rawtherapee and open with the RAW editor (from with Shotwell).  I applied the BW1 style, fixed the barrel distortion manually, activated the noise reduction with the default settings (which didn't make a big difference) and increased the already active sharpness level. Did a few cycles of "export from Rawtherapee - let Shotwell import the new JPG - merge events" to compare.  I ended up with a smaller JPEG than the camera one, less contrasty too (the in-camera settings are pretty high on sharpening and contrast, I would need to increase that in Rawtherapee to get a closer match and I guess the file size would increase too).

Unfortunately the integration between Rawtherapee and Shotwell stops there.  None of the metadata can be shared (as Shotwell doesn't write that to RAW files).  Rawtherapee has a metadata editor but I can't make it spit out the hierarchical keyword/tag format that Shotwell understands so it's all flat keywords.  For the rest I could title the photos through IPTC "Title" or "Legend" (Shotwell picks them in that order if specified).  But couldn't find a "Description" field to use in Rawtherapee and show in Shotwell (however if the JPEG contains one, maybe other apps might pick it but it wouldn't be visible in Shotwell unfortunately).

Darktable tests

Tried the version that comes with my current Ubuntu distro: 1.4.2

So far I've used

I liked

• the database (keep these RAW files organized!)
• the long list of powerful modules (though I only used the most basic ones so far)
• the interface in general (lighttable/darkroom)
• the metadata editor (IPTC tab) to set description title and keywords, save templates and apply them later, the tag list and its compatibility with Shotwell hierarchical tags (tag1|tag2|tag3), the starring.  I was able to produce JPEGs from within Darktable and import them into Shotwell without additional manual metadata tweaking.  Pretty nice on that front.

What I initially disliked but found a solution/workaround for

• DT doesn't (yet) support my Olympus XZ-10 RAW format (ORF) and my whishlisted OMD E-M10.  Fortunately this converts easily to DNG thanks to the kiwi DNG converter.  The resulting DNG file works fine with DT 1.4.2 so this is a minor concern for me now.
• the metadata editor can only be used from the lighttable, not from the darkroom view
• I found no key shortcut to "reject" (move to thrash) a photo from the darkroom view.  Clicking on the film strip works though. Then I found I could stay in lighttable view, make the thumbnails bigger (even display them one by one) where the shortcuts work (r, 12345, left/right, etc...) and this is much faster for preselecting images (compared to the darkroom view).
• I initially had a serious problem while testing with my DNG raw files from the Pentax K-r and also the DNG-converted ORFs from the XZ-10 (see above):  during editing everything looks fine but after exporting the JPG these photos are way too dark (like 1-3 stops darker after export).  I thought it might be due to Pentax's shadow/highlight in-camera settings.  I'll have to check that out.  The XZ-10 problems might be due to the DNG converter, or just the fact that this camera is not supported yet.It turns out that I somehow had an exotic color profile selected as the export profile (in the export box).  Turned this back to sRGB and it was fine.  The impending profile is reported by ExifTool as "Profile Description : Darktable linear Rec709 RGB".

backuppc - the ultimate automated backup solution?

That thing rocks!

Setup was moderately unsmooth as I needed to experiment a bit with the different options.  I have a  mix of Linux and Windows (different versions), laptops and desktops, DHCP on all of them.  I tried a bit of everything. The localhost (backup server) is backed up by the tar method (with help from visudo).  Other Linux hosts (or dual-boot linux+windows hosts) are backed up by rsync over ssh (with help from rsa certificates for authentication and a restriction to allow this only from the backup server's hostname).  Windows hosts are backed up by SMB and rather than giving administrator access I decided to create a local user on each of them and share only the parts of the disk that need back ups (like C:\Users) and give access to that user.  I'll have to rely on the Windows firewall to restrict this usage only from the backup server host name.

After a week or so, I was confident that this would work but I re-installed the whole thing to take advantage of LVM partitioning on my new 2TB SATA drive installed especially for this purpose in the backup server.  In my initial install I had partitioned that thing in the traditional way and I had a number of partitions on that drive for no good reason.

Now another week is almost over and I can say everything works as expected.  Backups are being taken (not necessarily daily, both clients and server need to be up and connected on the network at the same time of course, and backup is only attempted hourly).  I only got one email sent over that period (the day I re-installed the thing, before I was able to configure one of the SMB hosts).

I start the server manually when I feel we all need some good backup (if nobody else powered it before I do) and I added a crontab entry to shut it down at 04:00 AM (when backups are expected to be finished).  We'll see how this works but so far, so good and much better than my previous solutions.  I went back yesterday to sourceforge to give a nice rating to this project.  I will keep an eye on the 4.x version as it progresses.