FeministWiki:Server setup: Difference between revisions

(82 intermediate revisions by the same user not shown)

Line 1:

~~'''!!! WORK IN PROGRESS !!!'''~~

These are the steps required to set up a new FeministWiki Debian server.

== Initial setup of the new server ==

This section describes various initialization tasks for the new server that are independent of the old server.

=== Configure reverse DNS ===

~~These are~~ the ~~steps required~~ to ~~set up a~~ new ~~FeministWiki Debian~~ server.

In the settings of the VPS host (e.g. Strato AG), you can configure reverse-DNS for the IP address of the server. Set the FQDN for the IP address to {{C|feministwiki.org}}. It's good to do this early since it can take some time to propagate.

=== Make feministwiki.dev point to the new server ===

During setup and testing of the new server, we want to make it accessible under the '''feministwiki.dev''' domain. So change the {{C|A}} entry of the feministwiki.dev DNS settings to point to the IP address of the new server.

=== Update & upgrade ===

Line 7:

Line 17:

First of all, let's make sure the system is up to date.

apt-get update

apt-get upgrade

apt-get dist-upgrade

=== Install miscellaneous tools ===

Line 15:

Line 25:

Some of these are needed further down, some are just good to have.

apt-get install certbot

apt-get install automysqlbackup \

~~apt-get install~~ dnsutils

certbot \

~~apt-get install~~ git

dnsutils \

~~apt-get install~~ mg

emacs \

~~apt-get install~~ moreutils

git \

~~apt-get install~~ net-tools

mg \

~~apt-get install~~ nmap

moreutils \

~~apt-get install~~ software-properties-common

net-tools \

~~apt~~-~~get install tree~~

nmap \

rsync \

software-properties-common \

tree

=== Fetch scripts & config repo ===

Set up GitHub ssh access by copying the {{C|.ssh/id_rsa}} from the old server. After that:

cd ~

git clone git@github.com:FeministWiki/FeministWiki.git repo

cp -a repo/root/* repo/root/.??* .

sh repo/decrypt-pwd.sh

The decryption script will prompt you for a password the first time it's used. Enter the password stored in {{C|/root/pwd/meta}} on the old server.

=== Set up firewall ===

Line 29:

Line 53:

For now, block everything but SSH.

apt-get install ufw

ufw allow proto tcp to 0.0.0.0/0 port 22

ufw enable

=== ~~Install server components~~ ===

=== Enable extra repositories ===

We ~~will now install all the software used for the various FeministWiki services. But first we~~ might want to add some additional package repositories so we can use the latest version of some of the used software.

We might want to add some additional package repositories so we can use the latest version of some of the used software.

Backports:

Backports is always OK to add since the packages don't get priority over the stable ones:

echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main > /etc/apt/sources.list.d/backports.list

PHP repo '''only''' if a very new version is needed:

wget -O /etc/apt/trusted.gpg.d/sury-php.gpg https://packages.sury.org/php/apt.gpg

echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-php.list

MariaDB repo '''only''' if a very new version is needed:

wget https://mariadb.org/mariadb_release_signing_key.asc

apt-key add mariadb_release_signing_key.asc

rm mariadb_release_signing_key.asc

echo "deb http://mirror.23media.de/mariadb/repo/10.4/debian $(lsb_release -sc) main" > /etc/apt/sources.list.d/mariadb.list

=== Install server components ===

Now we can install all the software used for the various FeministWiki services:

~~Now we can~~ install ~~everything:~~

apt-get install apache2 \

dovecot-core \

dovecot-imapd \

dovecot-ldap \

dovecot-pop3d \

ejabberd \

fail2ban \

inspircd \

mailman \

mariadb-server \

opendkim \

postfix \

postfix-ldap \

slapd

~~apt-get install apache2~~

If any installation asks you for a password, remember that most passwords are found in {{C|/root/pwd}}.

~~apt-get install dovecot-core~~

~~apt-get install ejabberd # good candidate~~ for ~~backports~~

~~apt-get install fail2ban~~

~~apt-get install mariadb-server~~

~~apt-get install opendkim~~

~~apt-get install php7~~.~~4 # or whatever version we're on~~

~~apt-get install postfix~~

~~apt-get install slapd~~

Example for installing ejabberd from backports instead:

apt-get install ejabberd/$(lsb_release -sc)-backports

apt-get install ejabberd/$(lsb_release -sc)-backports # e.g. ejabberd/buster-backports

=== Install PHP and modules ===

~~=== Fetch scripts & config repo ===~~

This should really be part of the last section, but due to the sheer number of PHP modules we want to install, it's in its own section:

~~Set up GitHub ssh access by copying~~ the ~~<code>.ssh/id_rsa</code> from~~ the ~~old server. After that~~:

~~cd ~~~

~~git clone git@github.com:FeministWiki/FeministWiki.git repo~~

~~cp -a repo/root/* .~~

~~openssl aes-256-cbc -d -md sha512 -pbkdf2 -iter 100000 -in repo/pwd.enc -out - | tar -xzf-~~

~~The <code>openssl</code> decryption command will prompt you for a password~~. ~~Enter the password stored in <code>/root/pwd/meta</code> on the old server.~~

php_version=7.4 # or whatever version we're on

apt-get install php${php_version} \

php${php_version}-apcu \

php${php_version}-bcmath \

php${php_version}-cli \

php${php_version}-ctype \

php${php_version}-curl \

php${php_version}-gd \

php${php_version}-gmp \

php${php_version}-iconv \

php${php_version}-imagick \

php${php_version}-intl \

php${php_version}-json \

php${php_version}-ldap \

php${php_version}-mbstring \

php${php_version}-mysql \

php${php_version}-opcache \

php${php_version}-readline \

php${php_version}-xml \

php${php_version}-zip

=== Put config files in place ===

~~TODO~~

The principle is simple: take all the config files from {{C|/root/repo/etc}} and put them where they belong in {{C|/etc}}. However, since a new server might mean much newer software, it's possible that some config files aren't compatible anymore, or that some new sensible defaults might be overwritten by the old config. Sadly figuring out these incompatibilities is a manual process: compare the new default config with the old default config and to our current config, to figure out what our new config should look like.

There's a number of things important to remember however:

* After copying in the new {{C|/etc/aliases}} file, run {{C|newaliases}} for the changes to take effect

* After populating {{C|/etc/letsencrypt/renewal-hooks}}, remember to {{C|chmod +x}} all the scripts

* Likewise, don't forget {{C|chmod +x}} for <code>/etc/cron.{hourly,daily,weekly,monthly}</code> and {{C|/etc/boot.d}}

=== Enable Apache modules, config, and sites ===

We need a number of Apache modules to be enabled which might not be enabled by default:

a2enmod expires

a2enmod headers

a2enmod macro

a2enmod rewrite

a2enmod ssl

a2enconf 99-feministwiki

a2ensite 000-wiki

a2ensite account

a2ensite blogs

a2ensite chat

a2ensite files

a2ensite forum

a2ensite mail

a2ensite xmpp

=== Create vmail user ===

groupadd -g 5000 vmail

useradd -u 5000 -g vmail -s /usr/sbin/nologin -d /home/vmail -m vmail

=== Initialize LetsEncrypt ===

~~The first run of Certbot has to be interactive~~, ~~so we will do that now~~. Since ~~the~~ DNS entries still point to the old server, we can't get a cert for the real domains. For now, make '''feministwiki.dev''' ~~point~~ to the ~~IP address~~ of the new server, ~~then proceed with~~ the ~~commands below~~.

First, initialize the certbot configuration:

certbot register -n --agree-tos -m technician@feministwiki.org

Since various DNS entries still point to the old server, we can't get a cert for the real domains yet. For now, just get one for feministwiki.dev:

ufw allow 80

letsencrypt-refresh --dev-only

ufw delete allow 80

Our {{C|letsencrypt-refresh}} script makes sure that the cert files are found in {{C|/etc/fw-certs}} and that the private key and cert-and-key bundle are owned by the "ssl-cert" group and are readable by group members. A number of users have to be added to this group so they can read said files:

adduser ejabberd ssl-cert

adduser irc ssl-cert

== Copying over live data ==

We want to make a first run of this copy process purely for testing purposes. Note that although some of the steps described in this section take a long time to finish, they can be done in parallel.

=== LDAP databases ===

Stop the LDAP server and delete the existing configuration and data '''on the new server (careful!)''':

# Commands to run on the NEW (fresh) server:

systemctl stop slapd

rm -r /etc/ldap/slapd.d/*

rm /var/lib/ldap/data.mdb

Then copy over the config and data by running these commands from the old server:

slapcat -n 0 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d'

slapcat -n 1 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 1'

==== Breaking changes in OpenLDAP ====

There might be incompatible changes between OpenLDAP (aka {{C|slapd}}) versions which require manual editing of the {{C|slapcat}} output before it's read in on the new server with {{C|slapadd}}.

Here's one example that occurs when updating from OpenLDAP 2.4.42 or earlier to 2.4.43 or later: the ppolicy overlay has a new attribute in the newer version, so if you simply run the commands above, the first one (the one that copies the config database) will produce the following error message:

User Schema load failed for attribute "pwdMaxRecordedFailure". Error code 17: attribute type undefined

The solution is as follows:

# On the new server, open {{C|/etc/ldap/schema/ppolicy.ldif}} and search for {{C|pwdMaxRecordedFailure}}. You will note that there is a {{C|olcAttributeTypes: ...}} entry that defines it, and also it's listed in the {{C|MAY}} attributes block of the {{C|olcObjectClasses: ...}} entry that defines the {{C|pwdPolicy}} object class.

# On the old server, save the output of {{C|slapcat -n 0}} to a file, open the file, and search for the block where the {{C|ppolicy}} schema is defined. It should start with the line {{C|dn: cn={4}ppolicy,cn=schema,cn=config}} (the {{C|{4}}} part might contain a different integer, that's OK). There, note that the {{C|olcAttributeTypes: ...}} entry for {{C|pwdMaxRecordedFailure}} is missing, and also it's not listed in the {{C|MAY}} list of the {{C|pwdPolicy}} object class definition. Copy over the attribute type definition from the {{C|ppolicy.ldif}} file on the new server, and amend the {{C|MAY}} list to include it.

The above is explained only for instructive purposes, since this particular fix will already have been applied by the time someone reads this guide. It's meant to give you an idea as to how backwards incompatible changes in OpenLDAP schema files can be amended when migrating to a newer version. (Also, no such clear explanation of the fix seems to be found anywhere on the web, so maybe someone who searches the error message above will come upon this guide and be happy!)

=== Contents of /var/www ===

This is very simple but takes a lot of time to finish. '''Run it from the old server:'''

rsync -az --delete /var/www/ root@feministwiki.dev:/var/www

Note that the trailing slash in {{C|/var/www/}} is important; if not provided, it will copy the directory to {{C|/var/www/www}} on the new server.

=== SQL databases ===

Run the following command from the old server:

mysqldump -u root -p"$(cat /root/pwd/mysql)" \

--add-drop-database \

--databases blogs \

feministfiles \

feministforum \

feministmail \

feministwiki \

feministwiki_de \

feministwiki_es \

feministwiki_it \

feministwiki_pt \

fff \

| gzip | ssh root@feministwiki.dev 'gunzip | /root/bin/sql'

You can use the {{C|show databases;}} command in the SQL console to make sure that the list of databases is complete. Unfortunately they have to be listed manually, because using the {{C|--all-databases}} option includes system databases that we don't want to copy.

=== Emails ===

This is a simple one. Run this command from the old server:

rsync -az --delete /home/vmail/ root@feministwiki.dev:/home/vmail

Note that the trailing slash in {{C|/home/vmail/}} is important.

=== Mailman data ===

GNU Mailman uses a filesystem-based "database" so we can transfer over its data as follows; run this from the old server:

cd /var/lib/mailman

rsync -az --delete archives data lists root@feministwiki.dev:/var/lib/mailman

And then this on the new server:

check_perms -f

The {{C|check_perms}} command, which is part of GNU Mailman, will take care of fixing file ownership and permissions.

== Recreate SQL users ==

If the versions of MariaDB on the old and new server are compatible enough, you might be able to dump the {{C|mysql.user}} table and import it on the new server, but it's safer to recreate the users from scratch. To do so, run this on the new server:

/root/bin/sql << EOF

create user blogs@localhost identified by '$(cat ~/pwd/mysql-blogs)';

grant all on blogs.* to blogs@localhost;

create user feministfiles@localhost identified by '$(cat ~/pwd/mysql-files)';

grant all on feministfiles.* to feministfiles@localhost;

create user feministforum@localhost identified by '$(cat ~/pwd/mysql-forum)';

grant all on feministforum.* to feministforum@localhost;

create user feministmail@localhost identified by '$(cat ~/pwd/mysql-mail)';

grant all on feministmail.* to feministmail@localhost;

create user feministwiki@localhost identified by '$(cat ~/pwd/mysql-wiki)';

grant all on feministwiki.* to feministwiki@localhost;

grant all on feministwiki_de.* to feministwiki@localhost;

grant all on feministwiki_es.* to feministwiki@localhost;

grant all on feministwiki_it.* to feministwiki@localhost;

grant all on feministwiki_pt.* to feministwiki@localhost;

create user fff@localhost identified by '$(cat ~/pwd/mysql-fff)';

grant all on fff.* to fff@localhost;

EOF

== Test ==

It's important to test the new server to make sure everything works well!

=== Reboot ===

We could restart a lot of services manually to ensure they've read their new config, but it's easiest to just reboot. (The new server, obviously.)

=== Open ports ===

We need to open all the ports used by the various FeministWiki services:

for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777

do ufw allow proto tcp to 0.0.0.0/0 port $port

done

=== Test! ===

At this point you should test everything using the feministwiki.dev domain name.

Some things may not work because they're hard-coded to work as "feministwiki.org" and not under the "feministwiki.dev" name. This is a point of future improvement: all the services should be configured, if at all possible, in a way that they will work when invoked as feministwiki.dev just as well.

=== Deactivate again ===

After we're done testing, we can "deactivate" the new server again to prepare it for the final switch-over:

for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777

do ufw delete allow proto tcp to 0.0.0.0/0 port $port

done

systemctl stop apache2

~~certbot certonly --cert-name feministwiki.org~~ ~~# Yes, it should be .org and not .dev~~

systemctl stop dovecot

systemctl stop ejabberd

systemctl stop inspircd

systemctl stop mailman

systemctl stop postfix

systemctl stop slapd

~~Then follow the on-screen instructions of certbot. Use the "standalone" authentication method when asked, and enter feministwiki.dev as the sole domain.~~

== Finishing up ==

~~=== Stop~~ services on old server ~~===~~

'''Now, all services on the old server should be stopped, because we will begin the final transfer of live data.'''

~~'''At this point, all~~ services on the old server ~~should be stopped, because we will now begin to transfer all the actual data.'''~~

=== Stop services on the old server ===

~~=== Copy over LDAP databases ===~~

Stop all the services that interface with users and/or are responsible for modifying live data:

~~TODO~~

systemctl stop apache2

systemctl stop dovecot

systemctl stop ejabberd

systemctl stop inspircd

systemctl stop mailman

systemctl stop postfix

systemctl stop slapd

~~=== Copy over SQL databases ===~~

Close all the relevant ports just to be double-sure:

~~TODO~~

for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777

do ufw delete allow proto tcp to 0.0.0.0/0 port $port

done

=== Copy over ~~/var/www~~ ===

=== Copy over the live data one more time ===

~~TODO~~

'''Simply repeat the whole section ''Copying over live data''.'''

~~=== Open ports ===~~

The techniques and commands described above in the section ''Copying over live data'' are ''idempotent'', meaning you can simply repeat them and they will make sure that the new copy of the live data is fresh and doesn't leave any outdated data on the new server. For instance, the {{C|--delete}} argument to the {{C|rsync}} command and the {{C|--add-drop-database}} argument to the {{C|mysqldump}} command help to make sure of this.

~~We are finally ready to serve:~~

So just repeat the steps from that section exactly one more time.

~~for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777~~

=== Reboot the new server ===

~~do ufw allow proto tcp to 0.0.0.0/0 port $port~~

~~done~~

At this point ~~you might want~~ to ~~do some testing with the '''feministwiki.dev''' domain~~.

At this point we can reboot the new server again, to make sure all services are properly restarted.

=== ~~Reboot~~ ===

=== Open ports on the new server ===

~~We could restart a lot of services manually, but it's easiest to just reboot.~~

Now we can open the ports again on the new server:

~~reboot~~

for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777

do ufw allow proto tcp to 0.0.0.0/0 port $port

done

=== Update DNS entries ===

Line 144:

Line 384:

You only have to change three DNS entries, since most of the subdomains work via CNAME entries:

* The main ~~<code>~~A~~</code>~~ entry for ~~<code>~~@~~</code>~~ (self-reference i.e. feministwiki.org)

* The main {{C|A}} entry for {{C|@}} (self-reference i.e. feministwiki.org)

* The ~~<code>~~A~~</code>~~ entry for ~~<code>~~smtp~~</code>~~ since this is not allowed to be a CNAME

* The {{C|A}} entry for {{C|smtp}} since this is not allowed to be a CNAME

* The ~~<code>~~A~~</code>~~ entry for ~~<code>~~xmpp~~</code>~~ since this is not allowed to be a CNAME

* The {{C|A}} entry for {{C|xmpp}} since this is not allowed to be a CNAME

==== feministwiki.net, feministwiki.de, fem.wiki, fffrauen.de ====

For these, you only have to change the main ~~<code>~~A~~</code>~~ entry, since they don't use SMTP or XMPP.

For these, you only have to change the main {{C|A}} entry, since they don't use SMTP or XMPP.

=== Update the certificate ===

Run the ~~<code>~~letsencrypt-refresh~~</code>~~ script to get a new certificate which includes all our domain names, since we had started out with just feministwiki.dev.

Run the {{C|letsencrypt-refresh}} script to get a new certificate which includes all our domain names, since we had started out with just feministwiki.dev.

After this, everything should be functional. If not, it's time for some debugging!

@@ Line 1: / Line 1: @@
-'''!!! WORK IN PROGRESS !!!'''
+These are the steps required to set up a new FeministWiki Debian server.
+== Initial setup of the new server ==
+This section describes various initialization tasks for the new server that are independent of the old server.
+=== Configure reverse DNS ===
-These are the steps required to set up a new FeministWiki Debian server.
+In the settings of the VPS host (e.g. Strato AG), you can configure reverse-DNS for the IP address of the server.  Set the FQDN for the IP address to {{C|feministwiki.org}}.  It's good to do this early since it can take some time to propagate.
+=== Make feministwiki.dev point to the new server ===
+During setup and testing of the new server, we want to make it accessible under the '''feministwiki.dev''' domain.  So change the {{C|A}} entry of the feministwiki.dev DNS settings to point to the IP address of the new server.
 === Update & upgrade ===
@@ Line 7: / Line 17: @@
 First of all, let's make sure the system is up to date.
-  apt-get update
+ apt-get update
-  apt-get upgrade
+ apt-get upgrade
-  apt-get dist-upgrade
+ apt-get dist-upgrade
 === Install miscellaneous tools ===
@@ Line 15: / Line 25: @@
 Some of these are needed further down, some are just good to have.
-  apt-get install certbot
+ apt-get install automysqlbackup \
-  apt-get install dnsutils
+                 certbot \
-  apt-get install git
+                 dnsutils \
-  apt-get install mg
+                 emacs \
-  apt-get install moreutils
+                 git \
-  apt-get install net-tools
+                 mg \
-  apt-get install nmap
+                 moreutils \
-  apt-get install software-properties-common
+                 net-tools \
-   apt-get install tree
+                 nmap \
+                 rsync \
+                 software-properties-common \
+                 tree
+=== Fetch scripts & config repo ===
+Set up GitHub ssh access by copying the {{C|.ssh/id_rsa}} from the old server.  After that:
+ cd ~
+ git clone git@github.com:FeministWiki/FeministWiki.git repo
+ cp -a repo/root/* repo/root/.??* .
+ sh repo/decrypt-pwd.sh
+The decryption script will prompt you for a password the first time it's used.  Enter the password stored in {{C|/root/pwd/meta}} on the old server.
 === Set up firewall ===
@@ Line 29: / Line 53: @@
 For now, block everything but SSH.
-  apt-get install ufw
+ apt-get install ufw
-  ufw allow proto tcp to 0.0.0.0/0 port 22
+ ufw allow proto tcp to 0.0.0.0/0 port 22
-  ufw enable
+ ufw enable
-=== Install server components ===
+=== Enable extra repositories ===
-We will now install all the software used for the various FeministWiki services.  But first we might want to add some additional package repositories so we can use the latest version of some of the used software.
+We might want to add some additional package repositories so we can use the latest version of some of the used software.
-Backports:
+Backports is always OK to add since the packages don't get priority over the stable ones:
-  echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main > /etc/apt/sources.list.d/backports.list
+ echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main > /etc/apt/sources.list.d/backports.list
 PHP repo '''only''' if a very new version is needed:
-  wget -O /etc/apt/trusted.gpg.d/sury-php.gpg https://packages.sury.org/php/apt.gpg
+ wget -O /etc/apt/trusted.gpg.d/sury-php.gpg https://packages.sury.org/php/apt.gpg
-  echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-php.list
+ echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-php.list
 MariaDB repo '''only''' if a very new version is needed:
-  wget https://mariadb.org/mariadb_release_signing_key.asc
+ wget https://mariadb.org/mariadb_release_signing_key.asc
-  apt-key add mariadb_release_signing_key.asc
+ apt-key add mariadb_release_signing_key.asc
-  rm mariadb_release_signing_key.asc
+ rm mariadb_release_signing_key.asc
-  echo "deb http://mirror.23media.de/mariadb/repo/10.4/debian $(lsb_release -sc) main" > /etc/apt/sources.list.d/mariadb.list
+ echo "deb http://mirror.23media.de/mariadb/repo/10.4/debian $(lsb_release -sc) main" > /etc/apt/sources.list.d/mariadb.list
+=== Install server components ===
+Now we can install all the software used for the various FeministWiki services:
-Now we can install everything:
+ apt-get install apache2 \
+                 dovecot-core \
+                 dovecot-imapd \
+                 dovecot-ldap \
+                 dovecot-pop3d \
+                 ejabberd \
+                 fail2ban \
+                 inspircd \
+                 mailman \
+                 mariadb-server \
+                 opendkim \
+                 postfix \
+                 postfix-ldap \
+                 slapd
-  apt-get install apache2
+If any installation asks you for a password, remember that most passwords are found in {{C|/root/pwd}}.
-  apt-get install dovecot-core
-  apt-get install ejabberd # good candidate for backports
-  apt-get install fail2ban
-  apt-get install mariadb-server
-  apt-get install opendkim
-  apt-get install php7.4 # or whatever version we're on
-  apt-get install postfix
-  apt-get install slapd
 Example for installing ejabberd from backports instead:
-  apt-get install ejabberd/$(lsb_release -sc)-backports
+ apt-get install ejabberd/$(lsb_release -sc)-backports # e.g. ejabberd/buster-backports
+=== Install PHP and modules ===
-=== Fetch scripts & config repo ===
+This should really be part of the last section, but due to the sheer number of PHP modules we want to install, it's in its own section:
-Set up GitHub ssh access by copying the <code>.ssh/id_rsa</code> from the old server.  After that:
-  cd ~
-  git clone git@github.com:FeministWiki/FeministWiki.git repo
-  cp -a repo/root/* .
-  openssl aes-256-cbc -d -md sha512 -pbkdf2 -iter 100000 -in repo/pwd.enc -out - | tar -xzf-
-The <code>openssl</code> decryption command will prompt you for a password.  Enter the password stored in <code>/root/pwd/meta</code> on the old server.
+ php_version=7.4 # or whatever version we're on
+ apt-get install php${php_version} \
+                 php${php_version}-apcu \
+                 php${php_version}-bcmath \
+                 php${php_version}-cli \
+                 php${php_version}-ctype \
+                 php${php_version}-curl \
+                 php${php_version}-gd \
+                 php${php_version}-gmp \
+                 php${php_version}-iconv \
+                 php${php_version}-imagick \
+                 php${php_version}-intl \
+                 php${php_version}-json \
+                 php${php_version}-ldap \
+                 php${php_version}-mbstring \
+                 php${php_version}-mysql \
+                 php${php_version}-opcache \
+                 php${php_version}-readline \
+                 php${php_version}-xml \
+                 php${php_version}-zip
 === Put config files in place ===
-TODO
+The principle is simple: take all the config files from {{C|/root/repo/etc}} and put them where they belong in {{C|/etc}}.  However, since a new server might mean much newer software, it's possible that some config files aren't compatible anymore, or that some new sensible defaults might be overwritten by the old config.  Sadly figuring out these incompatibilities is a manual process: compare the new default config with the old default config and to our current config, to figure out what our new config should look like.
+There's a number of things important to remember however:
+* After copying in the new {{C|/etc/aliases}} file, run {{C|newaliases}} for the changes to take effect
+* After populating {{C|/etc/letsencrypt/renewal-hooks}}, remember to {{C|chmod +x}} all the scripts
+* Likewise, don't forget {{C|chmod +x}} for <code>/etc/cron.{hourly,daily,weekly,monthly}</code> and {{C|/etc/boot.d}}
+=== Enable Apache modules, config, and sites ===
+We need a number of Apache modules to be enabled which might not be enabled by default:
+ a2enmod expires
+ a2enmod headers
+ a2enmod macro
+ a2enmod rewrite
+ a2enmod ssl
+ a2enconf 99-feministwiki
+ a2ensite 000-wiki
+ a2ensite account
+ a2ensite blogs
+ a2ensite chat
+ a2ensite files
+ a2ensite forum
+ a2ensite mail
+ a2ensite xmpp
 === Create vmail user ===
-  groupadd -g 5000 vmail
+ groupadd -g 5000 vmail
-  useradd -u 5000 -g vmail -s /usr/sbin/nologin -d /home/vmail -m vmail
+ useradd -u 5000 -g vmail -s /usr/sbin/nologin -d /home/vmail -m vmail
 === Initialize LetsEncrypt ===
-The first run of Certbot has to be interactive, so we will do that now.  Since the DNS entries still point to the old server, we can't get a cert for the real domains.  For now, make '''feministwiki.dev''' point to the IP address of the new server, then proceed with the commands below.
+First, initialize the certbot configuration:
+ certbot register -n --agree-tos -m technician@feministwiki.org
+Since various DNS entries still point to the old server, we can't get a cert for the real domains yet.  For now, just get one for feministwiki.dev:
+ ufw allow 80
+ letsencrypt-refresh --dev-only
+ ufw delete allow 80
+Our {{C|letsencrypt-refresh}} script makes sure that the cert files are found in {{C|/etc/fw-certs}} and that the private key and cert-and-key bundle are owned by the "ssl-cert" group and are readable by group members.  A number of users have to be added to this group so they can read said files:
+ adduser ejabberd ssl-cert
+ adduser irc ssl-cert
+== Copying over live data ==
+We want to make a first run of this copy process purely for testing purposes.  Note that although some of the steps described in this section take a long time to finish, they can be done in parallel.
+=== LDAP databases ===
+Stop the LDAP server and delete the existing configuration and data '''on the new server (careful!)''':
+ # Commands to run on the NEW (fresh) server:
+ systemctl stop slapd
+ rm -r /etc/ldap/slapd.d/*
+ rm /var/lib/ldap/data.mdb
+Then copy over the config and data by running these commands from the old server:
+ slapcat -n 0 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d'
+ slapcat -n 1 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 1'
+==== Breaking changes in OpenLDAP ====
+There might be incompatible changes between OpenLDAP (aka {{C|slapd}}) versions which require manual editing of the {{C|slapcat}} output before it's read in on the new server with {{C|slapadd}}.
+Here's one example that occurs when updating from OpenLDAP 2.4.42 or earlier to 2.4.43 or later: the ppolicy overlay has a new attribute in the newer version, so if you simply run the commands above, the first one (the one that copies the config database) will produce the following error message:
+ User Schema load failed for attribute "pwdMaxRecordedFailure". Error code 17: attribute type undefined
+The solution is as follows:
+# On the new server, open {{C|/etc/ldap/schema/ppolicy.ldif}} and search for {{C|pwdMaxRecordedFailure}}.  You will note that there is a {{C|olcAttributeTypes: ...}} entry that defines it, and also it's listed in the {{C|MAY}} attributes block of the {{C|olcObjectClasses: ...}} entry that defines the {{C|pwdPolicy}} object class.
+# On the old server, save the output of {{C|slapcat -n 0}} to a file, open the file, and search for the block where the {{C|ppolicy}} schema is defined.  It should start with the line {{C|dn: cn={4}ppolicy,cn=schema,cn=config}} (the {{C|{4}}} part might contain a different integer, that's OK).  There, note that the {{C|olcAttributeTypes: ...}} entry for {{C|pwdMaxRecordedFailure}} is missing, and also it's not listed in the {{C|MAY}} list of the {{C|pwdPolicy}} object class definition.  Copy over the attribute type definition from the {{C|ppolicy.ldif}} file on the new server, and amend the {{C|MAY}} list to include it.
+The above is explained only for instructive purposes, since this particular fix will already have been applied by the time someone reads this guide.  It's meant to give you an idea as to how backwards incompatible changes in OpenLDAP schema files can be amended when migrating to a newer version.  (Also, no such clear explanation of the fix seems to be found anywhere on the web, so maybe someone who searches the error message above will come upon this guide and be happy!)
+=== Contents of /var/www ===
+This is very simple but takes a lot of time to finish.  '''Run it from the old server:'''
+ rsync -az --delete /var/www/ root@feministwiki.dev:/var/www
+Note that the trailing slash in {{C|/var/www/}} is important; if not provided, it will copy the directory to {{C|/var/www/www}} on the new server.
+=== SQL databases ===
+Run the following command from the old server:
+ mysqldump -u root -p"$(cat /root/pwd/mysql)" \
+   --add-drop-database \
+   --databases blogs \
+               feministfiles \
+               feministforum \
+               feministmail \
+               feministwiki \
+               feministwiki_de \
+               feministwiki_es \
+               feministwiki_it \
+               feministwiki_pt \
+               fff \
+   | gzip | ssh root@feministwiki.dev 'gunzip | /root/bin/sql'
+You can use the {{C|show databases;}} command in the SQL console to make sure that the list of databases is complete.  Unfortunately they have to be listed manually, because using the {{C|--all-databases}} option includes system databases that we don't want to copy.
+=== Emails ===
+This is a simple one.  Run this command from the old server:
+ rsync -az --delete /home/vmail/ root@feministwiki.dev:/home/vmail
+Note that the trailing slash in {{C|/home/vmail/}} is important.
+=== Mailman data ===
+GNU Mailman uses a filesystem-based "database" so we can transfer over its data as follows; run this from the old server:
+ cd /var/lib/mailman
+ rsync -az --delete archives data lists root@feministwiki.dev:/var/lib/mailman
+And then this on the new server:
+ check_perms -f
+The {{C|check_perms}} command, which is part of GNU Mailman, will take care of fixing file ownership and permissions.
+== Recreate SQL users ==
+If the versions of MariaDB on the old and new server are compatible enough, you might be able to dump the {{C|mysql.user}} table and import it on the new server, but it's safer to recreate the users from scratch.  To do so, run this on the new server:
+ /root/bin/sql << EOF
+ create user blogs@localhost identified by '$(cat ~/pwd/mysql-blogs)';
+ grant all on blogs.* to blogs@localhost;
+ create user feministfiles@localhost identified by '$(cat ~/pwd/mysql-files)';
+ grant all on feministfiles.* to feministfiles@localhost;
+ create user feministforum@localhost identified by '$(cat ~/pwd/mysql-forum)';
+ grant all on feministforum.* to feministforum@localhost;
+ create user feministmail@localhost identified by '$(cat ~/pwd/mysql-mail)';
+ grant all on feministmail.* to feministmail@localhost;
+ create user feministwiki@localhost identified by '$(cat ~/pwd/mysql-wiki)';
+ grant all on feministwiki.* to feministwiki@localhost;
+ grant all on feministwiki_de.* to feministwiki@localhost;
+ grant all on feministwiki_es.* to feministwiki@localhost;
+ grant all on feministwiki_it.* to feministwiki@localhost;
+ grant all on feministwiki_pt.* to feministwiki@localhost;
+ create user fff@localhost identified by '$(cat ~/pwd/mysql-fff)';
+ grant all on fff.* to fff@localhost;
+ EOF
+== Test ==
+It's important to test the new server to make sure everything works well!
+=== Reboot ===
+We could restart a lot of services manually to ensure they've read their new config, but it's easiest to just reboot.  (The new server, obviously.)
+=== Open ports ===
+We need to open all the ports used by the various FeministWiki services:
+ for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
+ do ufw allow proto tcp to 0.0.0.0/0 port $port
+ done
+=== Test! ===
+At this point you should test everything using the feministwiki.dev domain name.
+Some things may not work because they're hard-coded to work as "feministwiki.org" and not under the "feministwiki.dev" name.  This is a point of future improvement: all the services should be configured, if at all possible, in a way that they will work when invoked as feministwiki.dev just as well.
+=== Deactivate again ===
+After we're done testing, we can "deactivate" the new server again to prepare it for the final switch-over:
+ for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
+ do ufw delete allow proto tcp to 0.0.0.0/0 port $port
+ done
   systemctl stop apache2
-  certbot certonly --cert-name feministwiki.org  # Yes, it should be .org and not .dev
+  systemctl stop dovecot
+  systemctl stop ejabberd
+ systemctl stop inspircd
+ systemctl stop mailman
+ systemctl stop postfix
+ systemctl stop slapd
-Then follow the on-screen instructions of certbot.  Use the "standalone" authentication method when asked, and enter feministwiki.dev as the sole domain.
+== Finishing up ==
-=== Stop services on old server ===
+'''Now, all services on the old server should be stopped, because we will begin the final transfer of live data.'''
-'''At this point, all services on the old server should be stopped, because we will now begin to transfer all the actual data.'''
+=== Stop services on the old server ===
-=== Copy over LDAP databases ===
+Stop all the services that interface with users and/or are responsible for modifying live data:
-TODO
+ systemctl stop apache2
+ systemctl stop dovecot
+ systemctl stop ejabberd
+ systemctl stop inspircd
+ systemctl stop mailman
+ systemctl stop postfix
+ systemctl stop slapd
-=== Copy over SQL databases ===
+Close all the relevant ports just to be double-sure:
-TODO
+ for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
+ do ufw delete allow proto tcp to 0.0.0.0/0 port $port
+ done
-=== Copy over /var/www ===
+=== Copy over the live data one more time ===
-TODO
+'''Simply repeat the whole section ''Copying over live data''.'''
-=== Open ports ===
+The techniques and commands described above in the section ''Copying over live data'' are ''idempotent'', meaning you can simply repeat them and they will make sure that the new copy of the live data is fresh and doesn't leave any outdated data on the new server.  For instance, the {{C|--delete}} argument to the {{C|rsync}} command and the {{C|--add-drop-database}} argument to the {{C|mysqldump}} command help to make sure of this.
-We are finally ready to serve:
+So just repeat the steps from that section exactly one more time.
-  for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
+=== Reboot the new server ===
-  do ufw allow proto tcp to 0.0.0.0/0 port $port
-  done
-At this point you might want to do some testing with the '''feministwiki.dev''' domain.
+At this point we can reboot the new server again, to make sure all services are properly restarted.
-=== Reboot ===
+=== Open ports on the new server ===
-We could restart a lot of services manually, but it's easiest to just reboot.
+Now we can open the ports again on the new server:
-  reboot
+ for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
+ do ufw allow proto tcp to 0.0.0.0/0 port $port
+ done
 === Update DNS entries ===
@@ Line 144: / Line 384: @@
 You only have to change three DNS entries, since most of the subdomains work via CNAME entries:
-* The main <code>A</code> entry for <code>@</code> (self-reference i.e. feministwiki.org)
+* The main {{C|A}} entry for {{C|@}} (self-reference i.e. feministwiki.org)
-* The <code>A</code> entry for <code>smtp</code> since this is not allowed to be a CNAME
+* The {{C|A}} entry for {{C|smtp}} since this is not allowed to be a CNAME
-* The <code>A</code> entry for <code>xmpp</code> since this is not allowed to be a CNAME
+* The {{C|A}} entry for {{C|xmpp}} since this is not allowed to be a CNAME
 ==== feministwiki.net, feministwiki.de, fem.wiki, fffrauen.de ====
-For these, you only have to change the main <code>A</code> entry, since they don't use SMTP or XMPP.
+For these, you only have to change the main {{C|A}} entry, since they don't use SMTP or XMPP.
 === Update the certificate ===
-Run the <code>letsencrypt-refresh</code> script to get a new certificate which includes all our domain names, since we had started out with just feministwiki.dev.
+Run the {{C|letsencrypt-refresh}} script to get a new certificate which includes all our domain names, since we had started out with just feministwiki.dev.
+After this, everything should be functional.  If not, it's time for some debugging!