Apache HTTPS certificate and renewal

In November last year I got my first server certificate (from letsencrypt.org).  So it's going to expire next week and it's really time to have a look at renewing it.
Turns out, I got the first certificate with certbot certonly, and it was pretty easy to get it renewed with certbot renew
Apache needed a restart to use the new certificate but now it's fine, and good to somewhere in early May.
I will have a look at automating the renewal with cron.

Piwigo on server

So I'm reinstalling Piwigo and stuff on my little server this time.

Software

sudo tasksel

Select LAMP server, OK -> install 28 packages or so.  Test: http://kid2.lan/ (OK: apache default page).

sudo apt install php7.2-mbstring php7.2-gd php7.2-mysql

Change the following in  /etc/php/7.2/apache2/php.ini :

upload_max_filesize = 20M
post_max_size = 20M
max_file_uploads = 99

Database

Create piwigo.sql script with the necessary CREATE DATABASE and so on.

mysql -u root -p < /home/luc/Documents/Config/piwogo.sql

Piwigo

cd /tmp && wget -O piwigo.zip \
http://piwigo.org/download/dlcounter.php?code=latest
unzip piwigo.zip
sudo mv piwigo /var/www/html/
sudo chown -R www-data:www-data /var/www/html/piwigo/
sudo chmod -R 755 /var/www/html/piwigo/

Apache configuration

sudo vi /etc/apache2/sites-available/piwigo.conf

Create this file as in previous blog entry, but on port 80 and without the SSL stuff for the moment.  We want to init Piwigo before, but you could do it after the SSL/HTTPS configuration too I guess.

Piwigo initialization

Goto http://kid2.lan/piwigo/ and the initialization script starts.  Test under HTTP for the time being.

SSL (HTTPS) configuration

Install certbot:

sudo apt install certbot

The rest of the instructions for using certbot it at :
https://desec.io/#!/en/docs/certbot

Apparently certbot needs curl to be installed, but I had to first upgrade urbackup-server to 2.2.11 to remove a dependency on libcurl3 from urbackup-server-2.2.7 (otherwise installing curl would have removed urbackup-server).
When the certificate was received, I just changed piwigo.conf to activate the SSL stuff, and also had to enable the Apache ssl module:

sudo vi /etc/apache2/sites-available/piwigo.conf
sudo a2enmod ssl

Piwigo is pretty good hey

yes it it.  Also, I played a bit with port 443 forwarding and https configuration in Apache, and it finally kind of works.

Dynamic DNS

I have used  https://desec.io/ after trying also no-ip before.  Works fine.  In the Proximus BBOX I did setup the dyndns client parameters:

Provider: Custom
User Name: myhostIchoseTheNameOf.dedyn.io (as on desec.io: my domain)
Password: (my password at desec.io)
Hostname: myhostIchoseTheNameOf.dedyn.io
Custom Service:
HTTPS: On
Server: update.dedyn.io
Port: 443
Request:  ?hostname=%5BmyhostIchoseTheNameOf.dedyn.io%5D&myip=%5BIP%5D
Update Interval: 3600

DHCP

The HTTPS server needs a fixed IP address so we can do port forwarding in the next step.  Just assign one in the DSL router dialogs

Port forwarding

Forward only port 443 to the fixed IP address you assigned to the HTTPS server

SSL

Obtaining the first certificate was OK after a lot of reading, and some trial and error.  I still have to document how this can be done through let's encrypt

Apache:

 <VirtualHost *:443>
        ServerAdmin admin@myserver.lan
        DocumentRoot /var/www/html/piwigo
        ServerName myhostIchoseTheNameOf.dedyn.io

        SSLEngine on
        SSLProtocol all -SSLv2
        SSLCertificateFile /etc/letsencrypt/live/myhostIchoseTheNameOf.dedyn.io/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/myhostIchoseTheNameOf.dedyn.io/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/myhostIchoseTheNameOf.dedyn.io/fullchain.pem
     <Directory /var/www/html/piwigo/>
        Options +FollowSymlinks
        AllowOverride All
        Require all granted
     </Directory>

     ErrorLog ${APACHE_LOG_DIR}/error.log
     CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

Piwigo photo gallery test

So I followed instructions, mainly from this site.
PHP 7.2 is available in Ubuntu now so I used the repositories instead of a PPA to install it.  In addition I had to install the following optional php modules:

• sudo apt install php7.2-gd
• sudo apt install php7.2-mysql
• sudo apt install php7.2-mbstring
• ???

The proceeded to finish the install using the web interface.  Worked OK, uploaded a bunch of photos.  So far so good, starting to test.  Idelly this should provide:

• Easy bulk upload from Darktable
• Use metadata when available in the JPEG file, and allow searching on them:
• title
• description
• tags (keywords)
• date taken
• geotagging
• ...
• Authenticate users in the web interface
• Allows users to view but also download to their local device (phone, pc)
• ?

So after some testing:

• Easy bulk upload from Darktable: not directly.  I thought it was possible but the feature request about this is not progressing much in 5 years.  Exporting to JPEG then uploading through the web interface is possible of course.
• Authenticate users in the web interface : Yes, looks fine
• Allows users to view but also download to their local device (phone, pc) : Yes, look fine
• Use metadata when available in the JPEG file, and allow searching on them: not out of the box.  There used to be a plugin (Advanced Metadata) to do that, but it's not been maintained and doesn't run on current Piwigo releases apparenty.  So this is a show stopper for me.

Conclusion: dropped user and database from mysql, left the rest of the LAMP stack installed, might get used by something else later on... then noticed that Shotwell has a working Publish interface to Piwigo so started all over again to test that.

Desktop is too slow, upgraded

So I bought another 10 year old tower to replace it.  Bought an HP xw4600 workstation, nice hardware for its time.  Q6600 quad core, 4gb ecc ram with 2 free dimm slots, relatively quiet, nvidia quadro 285 with dual monitor support, free drive bays, multi card reader, lots of PCI slots, USB, firewire and ps2 ports, raid sata controller with 2x250gb caviars, DVD burner.

Installed Lubuntu on one of the drives, added urbackup client, then photo apps.  Set up autofs as NFS client. Tested Darktable using NFS and local copies, it looks much faster than the old one.  Exports still taking time over NFS at least.

So I removed the 2tb home drive from kid3 and installed it in the HP. Worked likes treat, but I still need to do some user and group admin.  In the meantime I finished 45 photos in Darktable by working locally and found no performance issues even though YouTube was consuming CPU and ram at the same time.

Memory could be upgraded, need to find some pc6400e dimms as the non-ecc support seems to be limited in this motherboard.

I also need to find a couple sata cables to get the DVD drive working (I stole the DVD cables for the 2tb HDD).

My monitors work fine with the VGA adapter that came with it but I now need a HDMI to dvi-D cable to extend my laptop display.  Think I already have that somewhere.

Next steps are the completion of a first full backup, RAM expansion to at least 8gb and later will be to clean up the laptop home dir and fix the filesystem and dpkg issue, maybe a reinstall is needed.

I'll be left with two functional tower PCs with little or no RAM or disks.  Maybe I'll try to refurbish and donate them.

Desktop is too slow, current state and some benchmarks (1/x)

So my desktop computer is becoming too slow for even the lightest photo editing tasks.  Don't even try video.  Here's the current setup:

$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 15
model name    : Intel(R) Pentium(R) Dual  CPU  E2200  @ 2.20GHz
stepping    : 13
microcode    : 0xa4
cpu MHz        : 1290.420
cache size    : 1024 KB
physical id    : 0
siblings    : 2
core id        : 0
cpu cores    : 2
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti dtherm
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips    : 4399.87

Memory is ok (nobody was logged in):

$ cat /proc/meminfo | head -20
MemTotal:        4039364 kB
MemFree:         2716668 kB
MemAvailable:    3529372 kB
Buffers:           70428 kB
Cached:           899376 kB
SwapCached:            0 kB
Active:           612904 kB
Inactive:         482644 kB
Active(anon):     126648 kB
Inactive(anon):    12484 kB
Active(file):     486256 kB
Inactive(file):   470160 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4085756 kB
SwapFree:        4085756 kB
Dirty:              5460 kB

I'm testing the system disk performance with bonnie++ (this is new to me):

$ time bonnie++ -d /tmp -s 0 -n 100 -f
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Create------ --------Random Create--------
kid3                -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                100 34604  72 +++++ +++ 45834  69 36578  74 +++++ +++ 40212  63
Latency             23276us    1109us    1045us     366us      55us     136ms
1.97,1.97,kid3,1,1539800153,,,,,,,,,,,,,,,100,,,,,34604,72,+++++,+++,45834,69,36578,74,+++++,+++,40212,63,,,,,,,23276us,1109us,1045us,366us,55us,136ms

real    0m11,145s
user    0m0,483s
sys    0m7,505s

For reference, the same command on the local system disk on my (about 2x faster) laptop:

$ time bonnie++ -d /tmp -s 0 -n 100 -f
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Create------ --------Random Create--------
run12l              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                100 60110  55 +++++ +++ 85174  60 68694  63 +++++ +++ 80560  59
Latency               283us     525us    4572us    1961us      46us    5125us
1.97,1.97,run12l,1,1539794180,,,,,,,,,,,,,,,100,,,,,60110,55,+++++,+++,85174,60,68694,63,+++++,+++,80560,59,,,,,,,283us,525us,4572us,1961us,46us,5125us

real    0m6,009s
user    0m0,200s
sys    0m3,520s

Same command again but access to the desktop system disk through NFS from my laptop.  I used much smaller number of files because it's a lot slower that way:

$ time bonnie++ -d /net/kid3.lan/home/luc/tmp -s 0 -n 5 -f
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Create------ --------Random Create--------
run12l              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  5   569  10 +++++ +++  1595  13   571  10  3440  19  1562  13
Latency             20565us   36590us    1070us   10365us    1543us    3710us
1.97,1.97,run12l,1,1539795060,,,,,,,,,,,,,,,5,,,,,569,10,+++++,+++,1595,13,571,10,3440,19,1562,13,,,,,,,20565us,36590us,1070us,10365us,1543us,3710us

real    0m26,198s
user    0m0,098s
sys    0m2,987s

Boththe laptop and the desktop are directly linked through an old ethernet hub/router/switch D-LINK DSL-604T that does not support higher than 100Mbps):

run12l $ ethtool eno1|grep Speed
    Speed: 100Mb/s

kid3:~$ ethtool enp0s15|grep Speed
    Speed: 100Mb/s

Partitions/Disks on the desktop, sdb is the system disk and sda1 is a larger disk holding all the non-system data:

$ blkid
/dev/sda1: LABEL="HOME" UUID="501fb987-b347-4b94-a2ac-97d971e28664" UUID_SUB="42cc63c7-590c-4aaf-b850-9e797ed2969e" TYPE="btrfs" PARTUUID="859ce7cb-01"
/dev/sdb1: LABEL="PQSERVICE" UUID="04D2821BE49BED2A" TYPE="ntfs" PARTUUID="2bd2c32a-01"
/dev/sdb2: LABEL="ACER" UUID="E0D8E5D0D8E5A4CE" TYPE="ntfs" PARTUUID="2bd2c32a-02"
/dev/sdb3: LABEL="VIDEOS" UUID="A46020DA6020B4C8" TYPE="ntfs" PARTUUID="2bd2c32a-03"
/dev/sdb5: UUID="b140a7c9-071d-48e2-861e-44242b16fce0" TYPE="swap" PARTUUID="2bd2c32a-05"
/dev/sdb6: UUID="41402262-1350-461f-bed3-b0e8e73610b7" TYPE="ext4" PARTUUID="2bd2c32a-06"

$ mount|grep sd
/dev/sdb6 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sdb2 on /windows type fuseblk (ro,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
/dev/sdb3 on /videos type fuseblk (rw,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
/dev/sda1 on /home type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

kid2/kid3 : new disk situation

So now kid3 only has two remaining disks: 500GB for system(s) and 2TB for btrfs /home

The WD 2TB drive has been moved to kid2 and after twiddling with the BIOS boot order kid2 went online again and happily started a full backup of run12l.

So I have a little bit of time to fiddle with the btrfs again.  Here's the situation (in addition, there's still an unconnected HDD in kid2, don't know what it is).

Systems are on /dev/sdb : 149,1 GiB, 160041885696 octets, 312581808 secteurs
btrfs small part on /dev/sda1 : 596,2 GiB, 640135028736 octets, 1250263728 secteurs
btrfs large part on /dev/sdd : 1,8 TiB, 2000398934016 octets, 3907029168 secteurs
So /dev/sdc is the WD migrated from kid3:

Disque /dev/sdc : 1,8 TiB, 2000398934016 octets, 3907029168 secteurs
Unités : sectors of 1 * 512 = 512 octets
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x000ef301

Périphérique Amorçage     Start        Fin   Secteurs   Size Id Type
/dev/sdc1                  2048  976575284  976573237 465,7G 83 Linux
/dev/sdc2             976576512 3907029167 2930452656   1,4T 8e LVM Linux

So after:
sudo btrfs device add -f /dev/sdc1 /var/backup/urBackup/

I get:
Label: none  uuid: 4881245e-bed5-4526-b9ff-3b5ba39b283d
        Total devices 3 FS bytes used 1.10TiB
        devid    3 size 596.17GiB used 567.03GiB path /dev/sda1
        devid    4 size 1.82TiB used 567.03GiB path /dev/sdd
        devid    5 size 465.67GiB used 0.00B path /dev/sdc1

Then to free the small 640GB drive I can use:
sudo btrfs device remove /dev/sda1 /var/backup/urBackup/