« FeministWiki:Server setup » : différence entre les versions
(→Emails) |
|||
Ligne 212 : | Ligne 212 : | ||
This is very simple but takes a lot of time to finish. '''Run it from the old server:''' | This is very simple but takes a lot of time to finish. '''Run it from the old server:''' | ||
rsync - | rsync -az --delete /var/www/ root@feministwiki.dev:/var/www | ||
Note that the trailing slash in <code>/var/www/</code> is important; if not provided, it will copy the directory to <code>/var/www/www</code> on the new server. | Note that the trailing slash in <code>/var/www/</code> is important; if not provided, it will copy the directory to <code>/var/www/www</code> on the new server. |
Version du 20 août 2021 à 12:34
These are the steps required to set up a new FeministWiki Debian server.
Initial setup of the new server
This section describes various initialization tasks for the new server that are independent of the old server.
Make feministwiki.dev point to the new server
During setup and testing of the new server, we want to make it accessible under the feministwiki.dev domain. So change the A
entry of the feministwiki.dev DNS settings to point to the IP address of the new server.
Update & upgrade
First of all, let's make sure the system is up to date.
apt-get update apt-get upgrade apt-get dist-upgrade
Install miscellaneous tools
Some of these are needed further down, some are just good to have.
apt-get install automysqlbackup apt-get install certbot apt-get install dnsutils apt-get install emacs apt-get install git apt-get install mg apt-get install moreutils apt-get install net-tools apt-get install nmap apt-get install rsync apt-get install software-properties-common apt-get install tree
Fetch scripts & config repo
Set up GitHub ssh access by copying the .ssh/id_rsa
from the old server. After that:
cd ~ git clone git@github.com:FeministWiki/FeministWiki.git repo cp -a repo/root/* repo/root/.??* . sh repo/decrypt-pwd.sh
The decryption script will prompt you for a password the first time it's used. Enter the password stored in /root/pwd/meta
on the old server.
Set up firewall
For now, block everything but SSH.
apt-get install ufw ufw allow proto tcp to 0.0.0.0/0 port 22 ufw enable
Enable extra repositories
We might want to add some additional package repositories so we can use the latest version of some of the used software.
Backports is always OK to add since the packages don't get priority over the stable ones:
echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main > /etc/apt/sources.list.d/backports.list
PHP repo only if a very new version is needed:
wget -O /etc/apt/trusted.gpg.d/sury-php.gpg https://packages.sury.org/php/apt.gpg echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-php.list
MariaDB repo only if a very new version is needed:
wget https://mariadb.org/mariadb_release_signing_key.asc apt-key add mariadb_release_signing_key.asc rm mariadb_release_signing_key.asc echo "deb http://mirror.23media.de/mariadb/repo/10.4/debian $(lsb_release -sc) main" > /etc/apt/sources.list.d/mariadb.list
Install server components
Now we can install all the software used for the various FeministWiki services:
apt-get install apache2 apt-get install dovecot-core apt-get install dovecot-imapd apt-get install dovecot-ldap apt-get install dovecot-pop3d apt-get install ejabberd # good candidate for backports apt-get install fail2ban apt-get install inspircd apt-get install mailman apt-get install mariadb-server apt-get install opendkim apt-get install postfix apt-get install postfix-ldap apt-get install slapd
If any installation asks you for a password, remember that most passwords are found in /root/pwd
.
Example for installing ejabberd from backports instead:
apt-get install ejabberd/$(lsb_release -sc)-backports # e.g. ejabberd/buster-backports
Install PHP and modules
This should really be part of the last section, but due to the sheer number of PHP modules we want to install, it's in its own section:
php_version=7.4 # or whatever version we're on apt-get install php${php_version} apt-get install php${php_version}-apcu apt-get install php${php_version}-bcmath apt-get install php${php_version}-cli apt-get install php${php_version}-ctype apt-get install php${php_version}-curl apt-get install php${php_version}-gd apt-get install php${php_version}-gmp apt-get install php${php_version}-iconv apt-get install php${php_version}-imagick apt-get install php${php_version}-intl apt-get install php${php_version}-json apt-get install php${php_version}-ldap apt-get install php${php_version}-mbstring apt-get install php${php_version}-mysql apt-get install php${php_version}-opcache apt-get install php${php_version}-readline apt-get install php${php_version}-xml apt-get install php${php_version}-zip
Put config files in place
The principle is simple: take all the config files from /root/repo/etc
and put them where they belong in /etc
. However, since a new server might mean much newer software, it's possible that some config files aren't compatible anymore, or that some new sensible defaults might be overwritten by the old config. Sadly figuring out these incompatibilities is a manual process: compare the new default config with the old default config and to our current config, to figure out what our new config should look like.
There's a number of things important to remember however:
- After copying in the new
/etc/aliases
file, runnewaliases
for the changes to take effect - After populating
/etc/letsencrypt/renewal-hooks
, remember tochmod +x
all the scripts
Enable Apache modules, config, and sites
We need a number of Apache modules to be enabled which might not be enabled by default:
a2enmod expires a2enmod headers a2enmod macro a2enmod rewrite a2enmod ssl a2enconf 99-feministwiki a2ensite 000-wiki a2ensite account a2ensite blogs a2ensite chat a2ensite files a2ensite forum a2ensite mail a2ensite xmpp
Create vmail user
groupadd -g 5000 vmail useradd -u 5000 -g vmail -s /usr/sbin/nologin -d /home/vmail -m vmail
Initialize LetsEncrypt
First, initialize the certbot configuration:
certbot register -n --agree-tos -m technician@feministwiki.org
Since various DNS entries still point to the old server, we can't get a cert for the real domains yet. For now, just get one for feministwiki.dev:
ufw allow 80 letsencrypt-refresh --dev-only ufw delete allow 80
Our letsencrypt-refresh
script makes sure that the cert files are found in /etc/fw-certs
and that the private key and cert-and-key bundle are owned by the "ssl-cert" group and are readable by group members. A number of users have to be added to this group so they can read said files:
adduser ejabberd ssl-cert adduser irc ssl-cert
Copying over live data
We want to make a first run of this copy process purely for testing purposes. Note that although some of the steps described in this section take a long time to finish, they can be done in parallel.
LDAP databases
Stop the LDAP server and delete the existing configuration and data on the new server (careful!):
# Commands to run on the NEW (fresh) server: systemctl stop slapd rm -r /etc/ldap/slapd.d/* rm /var/lib/ldap/data.mdb
Then copy over the config and data by running these commands from the old server:
slapcat -n 0 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d' slapcat -n 1 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 1'
Breaking changes in OpenLDAP
There might be incompatible changes between OpenLDAP (aka slapd
) versions which require manual editing of the slapcat
output before it's read in on the new server with slapadd
.
Here's one example that occurs when updating from OpenLDAP 2.4.42 or earlier to 2.4.43 or later: the ppolicy overlay has a new attribute in the newer version, so if you simply run the commands above, the first one (the one that copies the config database) will produce the following error message:
User Schema load failed for attribute "pwdMaxRecordedFailure". Error code 17: attribute type undefined
The solution is as follows:
- On the new server, open
/etc/ldap/schema/ppolicy.ldif
and search forpwdMaxRecordedFailure
. You will note that there is aolcAttributeTypes: ...
entry that defines it, and also it's listed in theMAY
attributes block of theolcObjectClasses: ...
entry that defines thepwdPolicy
object class. - On the old server, save the output of
slapcat -n 0
to a file, open the file, and search for the block where theppolicy
schema is defined. It should start with the linedn: cn={4}ppolicy,cn=schema,cn=config
(the{4}
part might contain a different integer, that's OK). There, note that theolcAttributeTypes: ...
entry forpwdMaxRecordedFailure
is missing, and also it's not listed in theMAY
list of thepwdPolicy
object class definition. Copy over the attribute type definition from theppolicy.ldif
file on the new server, and amend theMAY
list to include it.
The above is explained only for instructive purposes, since this particular fix will already have been applied by the time someone reads this guide. It's meant to give you an idea as to how backwards incompatible changes in OpenLDAP schema files can be amended when migrating to a newer version. (Also, no such clear explanation of the fix seems to be found anywhere on the web, so maybe someone who searches the error message above will come upon this guide and be happy!)
Contents of /var/www
This is very simple but takes a lot of time to finish. Run it from the old server:
rsync -az --delete /var/www/ root@feministwiki.dev:/var/www
Note that the trailing slash in /var/www/
is important; if not provided, it will copy the directory to /var/www/www
on the new server.
SQL databases
Run the following command from the old server:
mysqldump -u root -p"$(cat /root/pwd/mysql)" \ --add-drop-database \ --databases blogs \ feministfiles \ feministforum \ feministmail \ feministwiki \ feministwiki_de \ feministwiki_es \ feministwiki_it \ feministwiki_pt \ fff \ | gzip | ssh root@feministwiki.dev 'gunzip | /root/bin/sql'
You can use the show databases;
command in the SQL console to make sure that the list of databases is complete. Unfortunately they have to be listed manually, because using the --all-databases
option includes system databases that we don't want to copy.
Emails
This is a simple one. Run this command from the old server:
rsync -az --delete /home/vmail/ root@feministwiki.dev:/home/vmail
Note that the trailing slash in /home/vmail/
is important.
Mailman data
GNU Mailman uses a filesystem-based "database" so we can transfer over its data as follows; run this from the old server:
cd /var/lib/mailman tar -czf - archives data lists | ssh root@feministwiki.dev 'cd /var/lib/mailman && tar -xzf - && check_perms -f'
The check_perms
command, which is part of GNU Mailman, will take care of fixing the group ownership of the extracted files.
Recreate SQL users
If the versions of MariaDB on the old and new server are compatible enough, you might be able to dump the mysql.user
table and import it on the new server, but it's safer to recreate the users from scratch. To do so, run this on the new server:
/root/bin/sql << EOF create user blogs@localhost identified by '$(cat ~/pwd/mysql-blogs)'; grant all on blogs.* to blogs@localhost; create user feministfiles@localhost identified by '$(cat ~/pwd/mysql-files)'; grant all on feministfiles.* to feministfiles@localhost; create user feministforum@localhost identified by '$(cat ~/pwd/mysql-forum)'; grant all on feministforum.* to feministforum@localhost; create user feministmail@localhost identified by '$(cat ~/pwd/mysql-mail)'; grant all on feministmail.* to feministmail@localhost; create user feministwiki@localhost identified by '$(cat ~/pwd/mysql-wiki)'; grant all on feministwiki.* to feministwiki@localhost; create user fff@localhost identified by '$(cat ~/pwd/mysql-fff)'; grant all on fff.* to fff@localhost; EOF
Test
It's important to test the new server to make sure everything works well!
Reboot
We could restart a lot of services manually to ensure they've read their new config, but it's easiest to just reboot. (The new server, obviously.)
Open ports
We need to open all the ports used by the various FeministWiki services:
for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777 do ufw allow proto tcp to 0.0.0.0/0 port $port done
Test!
At this point you should test everything using the feministwiki.dev domain name.
Some things may not work because they're hard-coded to work as "feministwiki.org" and not under the "feministwiki.dev" name. This is a point of future improvement: all the services should be configured, if at all possible, in a way that they will work when invoked as feministwiki.dev just as well.
Finishing up
Now, all services on the old server should be stopped, because we will begin the final transfer of live data.
Stop services on the old server
Stop all the services that interface with users and/or are responsible for modifying live data:
systemctl stop apache2 systemctl stop dovecot systemctl stop ejabberd systemctl stop inspircd systemctl stop mailman systemctl stop postfix systemctl stop slapd
Close all the relevant ports just to be double-sure:
for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777 do ufw delete allow proto tcp to 0.0.0.0/0 port $port done
Copy over the live data one more time
The techniques and commands described above in the section Copying over live data are idempotent, meaning you can simply repeat them and they will make sure that the new copy of the live data is fresh and doesn't leave any outdated data on the new server. (For instance, the --delete
argument to the rsync
command and the --add-drop-database
argument to the mysqldump
command help to make sure of this.)
So in short, just repeat the steps from that section exactly one more time.
Update DNS entries
You have to change the configuration of the following domains:
- feministwiki.org
- feministwiki.net
- feministwiki.de
- fem.wiki
- fffrauen.de
feministwiki.org
You only have to change three DNS entries, since most of the subdomains work via CNAME entries:
- The main
A
entry for@
(self-reference i.e. feministwiki.org) - The
A
entry forsmtp
since this is not allowed to be a CNAME - The
A
entry forxmpp
since this is not allowed to be a CNAME
feministwiki.net, feministwiki.de, fem.wiki, fffrauen.de
For these, you only have to change the main A
entry, since they don't use SMTP or XMPP.
Update the certificate
Run the letsencrypt-refresh
script to get a new certificate which includes all our domain names, since we had started out with just feministwiki.dev.
After this, everything should be functional. If not, it's time for some debugging!