FeministWiki:Server setup

!!! WORK IN PROGRESS !!!

These are the steps required to set up a new FeministWiki Debian server.

Initial setup of the new server

This section describes various initialization tasks for the new server that are independent of the old server.

Make feministwiki.dev point to the new server

During setup and testing of the new server, we want to make it accessible under the feministwiki.dev domain. So change the A entry of the feministwiki.dev DNS settings to point to the IP address of the new server.

Update & upgrade

First of all, let's make sure the system is up to date.

apt-get update
apt-get upgrade
apt-get dist-upgrade

Install miscellaneous tools

Some of these are needed further down, some are just good to have.

apt-get install automysqlbackup
apt-get install certbot
apt-get install dnsutils
apt-get install emacs
apt-get install git
apt-get install mg
apt-get install moreutils
apt-get install net-tools
apt-get install nmap
apt-get install rsync
apt-get install software-properties-common
apt-get install tree

Fetch scripts & config repo

Set up GitHub ssh access by copying the .ssh/id_rsa from the old server. After that:

cd ~
git clone git@github.com:FeministWiki/FeministWiki.git repo
cp -a repo/root/* repo/root/.??* .
sh repo/decrypt-pwd.sh

The decryption script will prompt you for a password the first time it's used. Enter the password stored in /root/pwd/meta on the old server.

Set up firewall

For now, block everything but SSH.

apt-get install ufw
ufw allow proto tcp to 0.0.0.0/0 port 22
ufw enable

Enable extra repositories

We might want to add some additional package repositories so we can use the latest version of some of the used software.

Backports is always OK to add since the packages don't get priority over the stable ones:

echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main > /etc/apt/sources.list.d/backports.list

PHP repo only if a very new version is needed:

wget -O /etc/apt/trusted.gpg.d/sury-php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-php.list

MariaDB repo only if a very new version is needed:

wget https://mariadb.org/mariadb_release_signing_key.asc
apt-key add mariadb_release_signing_key.asc
rm mariadb_release_signing_key.asc
echo "deb http://mirror.23media.de/mariadb/repo/10.4/debian $(lsb_release -sc) main" > /etc/apt/sources.list.d/mariadb.list

Install server components

Now we can install all the software used for the various FeministWiki services:

php_version=7.4 # or whatever version we're on

apt-get install apache2
apt-get install dovecot-core
apt-get install dovecot-imapd
apt-get install dovecot-pop3d
apt-get install ejabberd # good candidate for backports
apt-get install fail2ban
apt-get install inspircd
apt-get install mailman
apt-get install mariadb-server
apt-get install opendkim
apt-get install php${php_version}
apt-get install php${php_version}-mbstring
apt-get install php${php_version}-xml
apt-get install postfix
apt-get install slapd

If any installation asks you for a password, remember that most passwords are found in /root/pwd.

Example for installing ejabberd from backports instead:

apt-get install ejabberd/$(lsb_release -sc)-backports # e.g. ejabberd/buster-backports

Enable Apache modules

We need a number of Apache modules to be enabled which might not be enabled by default:

a2enmod expires
a2enmod headers
a2enmod macro
a2enmod rewrite
a2enmod ssl

Create vmail user

groupadd -g 5000 vmail
useradd -u 5000 -g vmail -s /usr/sbin/nologin -d /home/vmail -m vmail

Put config files in place

The principle is simple: take all the config files from /root/repo/etc and put them where they belong in /etc. However, since a new server might mean much newer software, it's possible that some config files aren't compatible anymore, or that some new sensible defaults might be overwritten by the old config. Sadly figuring out these incompatibilities is a manual process: compare the new default config with the old default config and to our current config, to figure out what our new config should look like.

There's a number of things important to remember however:

• After copying in the new /etc/aliases file, run newaliases for the changes to take effect
• After populating /etc/letsencrypt/renewal-hooks, remember to chmod +x all the scripts

Initialize LetsEncrypt

First, initialize the certbot configuration:

certbot register -n --agree-tos -m technician@feministwiki.org

Since various DNS entries still point to the old server, we can't get a cert for the real domains yet. For now, just get one for feministwiki.dev:

ufw allow 80
letsencrypt-refresh --dev-only
ufw delete allow 80

Our letsencrypt-refresh script makes sure that the cert files are found in /etc/fw-certs and that the private key and cert-and-key bundle are owned by the "ssl-cert" group and are readable by group members. A number of users have to be added to this group so they can read said files:

adduser ejabberd ssl-cert
adduser irc ssl-cert

Copying over live data

We want to make a first run of this copy process purely for testing purposes, before shutting down the services on the old server and repeating it to ensure integrity of live data.

Copy over /var/www

This is very simple but takes a lot of time to finish. Run it from the old server:

rsync -a --delete /var/www/ root@feministwiki.dev:/var/www

Note that the trailing slash in /var/www/ is important; if not provided, it will copy the directory to /var/www/www on the new server.

While this command is running, you can do the next two steps in parallel.

Copy over SQL databases

Run the following command from the old server:

mysqldump -u root -p"$(cat /root/pwd/mysql)" \
  --add-drop-database \
  --databases blogs \
              feministblog \
              feministfiles \
              feministforum \
              feministmail \
              feministsocial \
              feministwiki \
              feministwiki_de \
              feministwiki_es \
              feministwiki_it \
              feministwiki_pt \
              fff \
  | ssh root@feministwiki.dev 'mysql -u root -p"$(cat /root/pwd/mysql)"'

You can use the show databases; command in the SQL console to make sure that the list of databases is complete. Unfortunately they have to be listed manually, because using the --all-databases option includes system databases that we don't want to copy.

While the above command is still running, you can do the next step in parallel.

Copy over LDAP databases

Stop the LDAP server and delete the existing configuration and data on the new server (careful!):

# Commands to run on the NEW (fresh) server:
systemctl stop slapd
rm -r /etc/ldap/slapd.d/*
rm /var/lib/ldap/data.mdb

Then copy over the config and data by running these commands from the old server:

slapcat -n 0 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d'
slapcat -n 1 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 1'

Note:

There might be incompatible changes between OpenLDAP (aka slapd) versions which require manual editing of the slapcat output before it's read in on the new server with slapadd.

Here's one example that occurs when updating from OpenLDAP 2.4.42 or earlier to 2.4.43 or later: the ppolicy overlay has a new attribute in the newer version, so if you simply run the commands above, the first one (the one that copies the config database) will produce the following error message:

User Schema load failed for attribute "pwdMaxRecordedFailure". Error code 17: attribute type undefined

The solution is as follows:

1. On the new server, open /etc/ldap/schema/ppolicy.ldif and search for pwdMaxRecordedFailure. You will note that there is a olcAttributeTypes: ... entry that defines it, and also it's listed in the MAY attributes block of the olcObjectClasses: ... entry that defines the pwdPolicy object class.
2. On the old server, save the output of slapcat -n 0 to a file, open the file, and search for the block where the ppolicy schema is defined. It should start with the line dn: cn={4}ppolicy,cn=schema,cn=config (the {4} part might contain a different integer, that's OK). There, note that the olcAttributeTypes: ... entry for pwdMaxRecordedFailure is missing, and also it's not listed in the MAY list of the pwdPolicy object class definition. Copy over the attribute type definition from the ppolicy.ldif file on the new server, and amend the MAY list to include it.

The above is explained only for instructive purposes, since this particular fix will already have been applied by the time someone reads this guide. It's meant to give you an idea as to how backwards incompatible changes in OpenLDAP schema files can be amended when migrating to a newer version. (Also, no such clear explanation of the fix seems to be found anywhere on the web, so maybe someone who searches the error message above will come upon this guide and be happy!)

Test

It's important to test the new server to make sure everything works well!

Reboot

We could restart a lot of services manually to ensure they've read their new config, but it's easiest to just reboot. (The new server, obviously.)

Test!

At this point you should test everything using the feministwiki.dev domain name.

Some things may not work because they're hard-coded to work as "feministwiki.org" and not under the "feministwiki.dev" name. This is a point of future improvement: all the services should be configured, if at all possible, in a way that they will work when invoked as feministwiki.dev just as well.

Finishing up

Now, all services on the old server should be stopped, because we will begin the final transfer of live data.

Stop services on the old server

Stop all the services that interface with users and/or are responsible for modifying live data:

systemctl stop apache2
systemctl stop dovecot
systemctl stop ejabberd
systemctl stop inspircd
systemctl stop mailman
systemctl stop mariadb
systemctl stop postfix
systemctl stop slapd

Close all the relevant ports just to be double-sure:

for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
do ufw delete allow proto tcp to 0.0.0.0/0 port $port
done

Copy over the live data one more time

The techniques and commands described above in the section Copying over live data are idempotent, meaning you can simply repeat them and they will make sure that the new copy of the live data is fresh and doesn't leave any outdated data on the new server. (For instance, the --delete argument to the rsync command and the --add-drop-database argument to the mysqldump command help to make sure of this.)

So in short, just repeat the steps from that section exactly one more time.

Update DNS entries

You have to change the configuration of the following domains:

• feministwiki.org
• feministwiki.net
• feministwiki.de
• fem.wiki
• fffrauen.de

feministwiki.org

You only have to change three DNS entries, since most of the subdomains work via CNAME entries:

• The main A entry for @ (self-reference i.e. feministwiki.org)
• The A entry for smtp since this is not allowed to be a CNAME
• The A entry for xmpp since this is not allowed to be a CNAME

feministwiki.net, feministwiki.de, fem.wiki, fffrauen.de

For these, you only have to change the main A entry, since they don't use SMTP or XMPP.

Open ports

We are almost ready to serve:

for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
do ufw allow proto tcp to 0.0.0.0/0 port $port
done

Update the certificate

Run the letsencrypt-refresh script to get a new certificate which includes all our domain names, since we had started out with just feministwiki.dev.

(This requires at least port 80 to be opened via the command in the last step.)

After this, everything should be functional. If not, it's time for some debugging!