urBackup episode 3

• Windows backup ended fine.  Other incremental backups have followed during the few times I boot this laptop under Windows 10 instead of Linux.  So far so good.  I've also tested a restore (one JPEG file), that went fine as well.
• I need to do something to save the config of the server itself I guess.  BackupPC provided the localhost host by default to do that, but the urBackup server doesn't have a default client on the server itself apparently; will check if I can install a Linux client on the same box and make that happen.
• The rtcwake script seems to work fine on kid2 too.  I enhanced it a bit to check for loadavg because it should not shutdown while the load average is anywhere above idle; so i don't need to do specific checks about urBackup processes and so forth
• One day I'll have to look at the email configuration too.  That's definitely broken by default for a home network
• The kid2 hardware looks like a good idea to start migrating to urBackup while not removing BackupPC immediately.  The only problem is I will need one more "big" hard disk before I can add any more clients to urBackup.

urBackup episode 2

I need more space for urBackup and no space for /home (I want to get rid of this home directory anyway). So unmounted /home, remounted it under /mnt, tried to move everything on the /home (in root filesystem), noticed that there wasn't enough room there because of all the cache and trash in there, did some cleanup then moved the rest... ok, I also moved the urBackup data on this 300Gb hard disk. Then changed /etc/fstab, remounted the disk under it's new location, restarted the urbackupsrv service... got some permission denied... changed the owner:group on the mount point, restarted the service again... that appears to have worked. I left the backup for my test Windows client running. It's quite big (lots of photos and videos) but hopefully there will be enough room this time.

urBackup proof of concept

So far I had not tested this solution because it wasn't available by default in the Ubuntu repository. However after my problems with RsyncD on Windows 10, I figured out that maybe another backup solution should be (at least) investigated, just in case backuppc wouldn't do well with Windows backups. I didn't want to go back to SMB backups if I could avoid it. urBackup looks like a nice, open source, polished solution. It is client-based (e.g. a client package has to be installed on each machine to be backed up) but at least it seems to support Windows very well (and I still have to test the Linux client which is essential for my needs). It promises to support backup compression and de-duplication although it is not clear what level of support (at least for compression it relies on the underlying OS and filesystem. Ext4 is not doing that AFAIK). So after adding a PPA, I was able to install a server for testing on kid2. The server configuration was pretty easy (email won't work but that's not urBackup's fault). I downloaded a client on run12l and set it up real quick, then the first backup started. I had to stop it before it could complete, will need to come back to it later.

Backuppc archives timeout problem + new archive HDD medium

So I took delivery of my new Maxtor M3 2Tb USB 3 drive yesterday.  Still the server only has USB 2 ports so I know this is going to be still slow until I can upgrade that.
However I hooked the M3 and it powered up and connected without problems.
fdisk showed a single partition (1.8Gb) formatted as HPFS/NTFS.  I moved the contents to /home/luc/Texts/Manuals or somewhere like that.  It contained some backup software and the documentation apparently.  Then I reformatted using ext4, and changed the partition type to Linux 0x83 (and removed the bootable flag).
Afterthat I used the UUID (from blkid) so add a new line to /etc/auto.removable and it just worked under autofs.  created the backuppc directory and will use that in a new backuppc archive host.
But before that I'd like to fill the iomega HDD with some recent backup archives so that I can move that out of site.  I noticed my previous attempts all failed after 20 hours exactly so after googling a bit I found out about the Client Timeout setting in Backuppc.  Changed this to 48 hours and started a new archive again, from kid3 backup.
To be continued...

Ubuntu 17.10

Last month I upgraded kid3 & run12l.  No issue, even got the latest Darktable etc so I didn't need to add any ppa so far.  Yesterday I also upgraded base so all OK for now.  I took a look inside (this is a Packard Bell B2501 BE).  Still spare room for disks (both SATA-several connectors free and one free IDE connector) and cards (no extension cards on the motherboard, using the on-board audio and video circuits).  There are 2 PCI slots and at least two PCIe slots (1x and longer).  There's also only 1 DIMM slot used and thus the second one is free.  Looks like this required DIMM DDR2 800.  Iirc the CPU is an E2200.  There's an unused HDD in that box, with only Linux partitions (/ swap /home probably) and it's about 200GB.  It still works (I connected it temporarily for testing) and seems to have data from 2014.  I think /home is empty though so nothing to save from there.  Could probably be used as recovery boot in case of failure of the main SATA HDD.

I ordered a PCIe 4-port USB3 card to install in this box in order to speed up the archives so I can start rolling out with my off-site backups.  I also bought a Maxtor M3 2Tb USB3 external hard disk (at Black Friday prices, 63€) but I still have to pick that up.

I might buy a used monitor screen to replace the one used by kid3 so I can move that one to base in order to troubleshoot a boot failure that I had when the IDE drive was connected (seen as /dev/sda).

Automounting removable USB disks for use as backuppc archive volumes

I want to be able to plug in a few (at least 2) removable HDD drives and use them to store my latest backups, so I can easily move these HDD drives offsite for additional protection against theft, fire or other catastrophic events.

Required software

 sudo apt install autofs

/etc/auto.master 

#

# Maps for external HDD etc 

# 

/var/backups/backuppcv /etc/auto.removable --timeout=2 

/etc/auto.removable

Currently I own two of these drives (Iomega is 1TB and Targa is 512MB).  Both are quite old but I plan to add a 2TB (or a couple of them) in the near future.  I did check the blkid from each of these drives then added these lines to my new auto.removable config file:
#
# Maps for external HDD etc
#
iomega    -fstype=ext2    :UUID=e66dd072-7a29-4021-8cea-6757bf3b566a
targa    -fstype=ext3    :UUID=bd524c4e-2c59-4685-9458-74b21a9c33cd

Activation

sudo service autofs restart

Permissions

mkdir /var/backups/backuppcv/iomega/backuppc

chown backuppc:backuppc /var/backups/backuppcv/iomega/backuppc

Usage with backuppc

sudo cat arc_iomega.pl
$Conf{XferMethod} = 'archive';
$Conf{ArchiveDest} = '/var/backups/backuppcv/iomega/backuppc';

Ooops involving backuppc

Tonight I wanted to remove postfix as I only need msmtp for sending email.  However this also accidentally removed the dependent package backuppc, including all configuration :-| as I had used "apt-get purge".

So I reinstalled backuppc (it also reinstalled postfix) and after setting up the htpasswd file then the localhost.pl file in /etc/backuppc (including the change to run tar under sudo) I was able to restore the previous /etc/backuppc/[hosts,*.pl] using backuppc itself.

I had to use "service backuppc restart" after rebooting kid2 as a test client before an incremental backup started OK.  Otherwise this looks fine.  No logs or backups seem to have been lost in the adventure.  This program is indeed more robust and foolproof than what I had thought.

I then restarted a full backup on f****-pc (I had previously given control to the backuppc user on this host to the user directory that needed backup).

If apache were also purged, the following is required on a blank Ubuntu/Apache2 install:
/etc/apache2/envvars:  run as backuppc as this is a dedicated apache2
/etc/apache2/sites-enabled/000-default.conf:    #ServerName www.example.com
/etc/apache2/mods-enabled$ sudo ln -s ../mods-available/cgi.load

Hardware Refresh, next steps

Hardware Refresh, next steps

kid2 recovery

kid2 didn't want to boot anymore (black screen) so I fiddled a little bit with HDD cabling until I could get into the BIOS setup.  The case still has 2GB PC2-6400 RAM (2x512Mb unbranded + 1x1GB Kingston with cooler).  One DIMM slot is still free so maybe I could add some more RAM in there.  This is a Gigabyte motherboard and I should still have the manual lying around somewhere.

After the fiddling, I tried booting the Vista partition (doesn't work, I knew that already) then the Vista recovery partition (did work but I cancelled the recovery as I want to backup data from the Windows partition first).

So I booted Xubuntu and did a local backup of the /windows/Users using tar --exclude .*<lots of unwanted stuff>.  The backup is too big for my biggest SD card so I need to put the PC back on the network somehow in order to transfer this.

kid3 backups

The full backup didn't complete last night due to /windows not being mounted.  The Linux partition was backed up though.  This results in a Partial full backup in backuppc.  Not good enough (even though there is currently no data on the Windows side).  I configured the NTFS partitions and permanently mounted the main one (that contains /windows/Users) then rescheduled the full backup for the next night.

kid3 Windows setup

I booted  Vista and logged in as the ACER (administrator) account with the password I had setup on the first boot.  Configured the network, refreshed Windows Defender, created 5 non-administrator accounts for the family (no passwords at this point).  My intention is to restore the Windows data from kid2 here before cleaning up kid2 and restoring Vista on kid2 (which likely will destroy that data).  I know I, for one, still have data on there and some family members have used kid2 in the past too, maybe under Vista.

kid3 Lubuntu setup

Did fiddle a little bit with a PCmanfm "action" or "custom command" so that the Upload2Flickr right-click will not start multiple instances when uploading several photos in one go.  This only involved using %F instead of %f but the fun part was to find where this *.desktop file was.
I urgently needed to work on *.chopro files and my scripts weren't working right anymore (some because of missing dependencies but then still some other problems including the need to recompile that old "chord" program.  I still had the necessary source package but I finally found out that a newer version had been developed and released as GPL and seems to be freshy maintained (under the name Chordii or chordii) so I just installed that from the repositories and tweaked my chordview script a little bit accordingly.  That worked fine.  I installed Libreoffice too to handle the lyrics output and found out that the recode latin1-utf8 seems to be useless now so I removed the call to recode.
Finally I had very little time left to test photo editing but I managed to edit and export 10 photos with a pretty good experience under Darktable 2.0.  The Lighttable is pretty fast when changing collections and scrolling (probably thanks to the additional RAM + the new cache in DT 2.0).  Exporting is still slowish but that's when it's OK to take a break from the editing so no problem.

Backuppc new backup client on kid3

1. Add the new host (kid3) using the web GUI
2. On kid3 (the new host):
1. sudo apt-get install openssh-server
3. On base (backuppc server) :
1. sudo apt-get update
2. sudo apt-get upgrade
3. cd /etc/backuppc
4. sudo cp -p kid2.pl kid3.pl
5. sudo ssh-copy-id luc@kid3
6. ssh luc@kid3 
7. (confirm key ID)
4. On kid3 (the new host):
1. cat ~luc/.ssh/authorized_keys >> ~root/.ssh/authorized_keys
2. chmod 750 ~root/.ssh/authorized_keys
5. On base (backuppc server) :
1. sudo ssh luc@kid3 
2. (confirm key ID)
6.  Start full backup using the web GUI

backuppc - the ultimate automated backup solution?

That thing rocks!

Setup was moderately unsmooth as I needed to experiment a bit with the different options.  I have a  mix of Linux and Windows (different versions), laptops and desktops, DHCP on all of them.  I tried a bit of everything. The localhost (backup server) is backed up by the tar method (with help from visudo).  Other Linux hosts (or dual-boot linux+windows hosts) are backed up by rsync over ssh (with help from rsa certificates for authentication and a restriction to allow this only from the backup server's hostname).  Windows hosts are backed up by SMB and rather than giving administrator access I decided to create a local user on each of them and share only the parts of the disk that need back ups (like C:\Users) and give access to that user.  I'll have to rely on the Windows firewall to restrict this usage only from the backup server host name.

After a week or so, I was confident that this would work but I re-installed the whole thing to take advantage of LVM partitioning on my new 2TB SATA drive installed especially for this purpose in the backup server.  In my initial install I had partitioned that thing in the traditional way and I had a number of partitions on that drive for no good reason.

Now another week is almost over and I can say everything works as expected.  Backups are being taken (not necessarily daily, both clients and server need to be up and connected on the network at the same time of course, and backup is only attempted hourly).  I only got one email sent over that period (the day I re-installed the thing, before I was able to configure one of the SMB hosts).

I start the server manually when I feel we all need some good backup (if nobody else powered it before I do) and I added a crontab entry to shut it down at 04:00 AM (when backups are expected to be finished).  We'll see how this works but so far, so good and much better than my previous solutions.  I went back yesterday to sourceforge to give a nice rating to this project.  I will keep an eye on the 4.x version as it progresses.