FeministWiki:Server setup: Difference between revisions

    From FeministWiki
    mNo edit summary
     
    (44 intermediate revisions by the same user not shown)
    Line 28: Line 28:
                     bsdutils \
                     bsdutils \
                     certbot \
                     certbot \
                    composer \
                     curl \
                     curl \
                     dnsutils \
                     dnsutils \
    Line 33: Line 34:
                     git \
                     git \
                     imagemagick \
                     imagemagick \
                    iotop \
                    ldap-utils \
                     mg \
                     mg \
                     moreutils \
                     moreutils \
    Line 90: Line 93:


      # Debian
      # Debian
      echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main > /etc/apt/sources.list.d/backports.list
    # May not be necessary; see if there's a commented out line in /etc/apt/sources.list that you can activate.
      echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main contrib non-free > /etc/apt/sources.list.d/backports.list
       
       
      # Ubuntu
      # Ubuntu
      echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-backports main universe > /etc/apt/sources.list.d/backports.list
      echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-backports main universe > /etc/apt/sources.list.d/backports.list


    PHP repo '''only''' if a very new version is needed:
    Usually you want up-to-date versions of Apache2, Nginx, and PHP.  Ondřej provides them:


      wget -O /etc/apt/trusted.gpg.d/sury-php.gpg https://packages.sury.org/php/apt.gpg
      # Debian
    echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-php.list
    for repo in apache2 nginx-mainline php
    do
      curl https://packages.sury.org/$repo/apt.gpg > /etc/apt/trusted.gpg.d/sury-$repo.gpg
      echo "deb https://packages.sury.org/$repo/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-$repo.list
    done
    # Ubuntu
    add-apt-repository ppa:ondrej/apache2
    add-apt-repository ppa:ondrej/nginx
    add-apt-repository ppa:ondrej/php
     
    It may be necessary to increase the priority of some repositories if you encounter errors about dependencies that can't be fulfilled, due to apt-get thinking it has to prioritize older versions of some packages.  This was last observed while trying to install {{C|nginx-extras}}, due to some library dependencies having the "epoch" part of the version number set in the regular Debian repository, but not set in the Sury repository.  The solution is to create a file in {{C|/etc/apt/preferences.d}} such as:
     
    Package: *
    Pin: origin packages.sury.org
    Pin-Priority: 700


    MariaDB repo '''only''' if a very new version is needed:
    Elasticsearch, if you want CirrusSearch for MediaWiki:


      wget https://mariadb.org/mariadb_release_signing_key.asc
      curl https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /etc/apt/trusted.gpg.d/elasticsearch.gpg
    apt-key add mariadb_release_signing_key.asc
      # As of January 2024, CirrusSearch only supports Elasticsearch 7.x
      rm mariadb_release_signing_key.asc
      echo 'deb https://artifacts.elastic.co/packages/7.x/apt stable main' > /etc/apt/sources.list.d/elastic.list
      echo "deb http://mirror.23media.de/mariadb/repo/10.4/debian $(lsb_release -sc) main" > /etc/apt/sources.list.d/mariadb.list


    === Create vmail user ===
    === Create vmail user ===
    Line 116: Line 134:
    Now we can install all the software used for the various FeministWiki services:
    Now we can install all the software used for the various FeministWiki services:


      apt-get install apache2 \
      apt-get install
                    dovecot-core \
        apache2 \
                    dovecot-imapd \
        dovecot-core \
                    dovecot-ldap \
        dovecot-imapd \
                    dovecot-pop3d \
        dovecot-ldap \
                    ejabberd \
        dovecot-pop3d \
                    fail2ban \
        ejabberd \
                    inspircd \
        elasticsearch \
                    mailman \
        fail2ban \
                    mariadb-server \
        inspircd \
                    opendkim \
        mariadb-server \
                    postfix \
        nginx-extras \
                    postfix-ldap \
        opendkim \
                    slapd
        postfix \
        postfix-ldap \
        slapd


    If any installation asks you for a password, remember that most passwords are found in {{C|/root/pwd}}.
    If any installation asks you for a password, remember that most passwords are found in {{C|/root/pwd}}.
    Line 135: Line 155:
    Example for installing ejabberd from backports instead:
    Example for installing ejabberd from backports instead:


      apt-get install ejabberd/$(lsb_release -sc)-backports # e.g. ejabberd/buster-backports
      apt-get install ejabberd/$(lsb_release -sc)-backports # e.g. ejabberd/bookworm-backports
     
    === Make sure Postfix can connect to OpenDKIM ===
     
    mkdir -p /var/spool/postfix/opendkim
    chown opendkim:opendkim /var/spool/postfix/opendkim
    adduser postfix opendkim


    === Install PHP and modules ===
    === Install PHP and modules ===
    Line 146: Line 172:
                     php${php_version}-apcu \
                     php${php_version}-apcu \
                     php${php_version}-bcmath \
                     php${php_version}-bcmath \
                    php${php_version}-bz2 \
                     php${php_version}-cli \
                     php${php_version}-cli \
                     php${php_version}-curl \
                     php${php_version}-curl \
    Line 160: Line 187:
                     php${php_version}-xml \
                     php${php_version}-xml \
                     php${php_version}-zip
                     php${php_version}-zip
    We also want {{C|php-luasandbox}}, which may not have a PHP version attached to the package name, in which case you'll have to make sure it supports the PHP version currently in use. If not, you can use the standalone Lua binary instead by setting {{C|$wgScribuntoDefaultEngine {{=}} 'luastandalone';}} in MediaWiki's {{C|LocalSettings.php}} configuration file.
    # See if this works first:
    apt-get install php${php_version}-luasandbox
    # Otherwise...
    apt-get install php-luasandbox
    # Check the package contents to see which PHP versions are supported
    dpkg -L php-luasandbox


    === Copy over certificates ===
    === Copy over certificates ===
    Line 165: Line 202:
    Copy over the certs from the old server:
    Copy over the certs from the old server:


      tar -czPf- /etc/fw-certs | ssh feministwiki.dev 'tar -xzPf-'
      # Run on old server
    rsync -avz /etc/feministwiki/certs feministwiki.dev:/etc/feministwiki/certs


    The {{C|/etc/fw-certs}} directory and its contents should be owned by the group {{C|ssl-cert}}.  Make sure this is the case on the new server after running the command above, since the group ID might be different on the new server.  If the group doesn't exist at all, just create it.
    The {{C|/etc/feministwiki/certs}} directory and its contents should be owned by the group {{C|ssl-cert}}.  Make sure this is the case on the new server after running the command above, since the group ID might be different on the new server.  If the group doesn't exist at all, just create it.


    Further, files in that directory which contain the private key ({{C|privkey.pem}} and {{C|bundle.pem}}) should only be readable by group members.  That is, their permission mode should be 640, displayed as {{C|-rw-r-----}} in the output of {{C|ls -l}}.  Make sure this really the case.
    Further, files in that directory which contain the private key ({{C|privkey.pem}} and {{C|bundle.pem}}) should only be readable by group members.  That is, their permission mode should be 640, displayed as {{C|-rw-r-----}} in the output of {{C|ls -l}}.  Make sure this really the case.
    Line 173: Line 211:
    Then, to allow certain services to read those files containing the private key, add them to the {{C|ssl-cert}} group:
    Then, to allow certain services to read those files containing the private key, add them to the {{C|ssl-cert}} group:


    # Run on new server
      adduser ejabberd ssl-cert
      adduser ejabberd ssl-cert
      adduser irc ssl-cert
      adduser irc ssl-cert
    Also copy over the certificates stored directly in {{C|/etc/letsencrypt}}:
    # Run on old server
    rsync -avz /etc/letsencrypt/{archive,live} feministwiki.dev:/etc/letsencrypt


    === Put config files in place ===
    === Put config files in place ===
    Line 183: Line 227:


    * Don't forget to revert the redactions of sensitive information.  Search files for {{C|[REDACTED]}}.
    * Don't forget to revert the redactions of sensitive information.  Search files for {{C|[REDACTED]}}.
    * After copying over the Postfix configuration, run {{C|newaliases}} to create the alias database files.
    * After copying {{C|/etc/aliases}}, run {{C|newaliases}} to create the alias database file.
    * You may also have to run {{C|postmap}} on some files, like {{C|/etc/postfix/sender_access}}.
    * After copying Postfix configuration, run {{C|postmap /etc/postfix/sender-access}} and {{C|postalias /etc/postfix/virtual-aliases}}.
    * After copying something into {{C|/etc/apache2/conf-available}}, don't forget to enable it via {{C|a2enconf}}.
    * Make sure the executable bit is set on all {{C|cron}} scripts, {{C|/etc/rc.local}}, and scripts in {{C|/etc/letsencrypt/renewal-hooks}}.
    * After populating {{C|/etc/letsencrypt/renewal-hooks}}, remember to {{C|chmod +x}} the scripts.
    * Likewise, don't forget {{C|chmod +x}} for <code>/etc/cron.{hourly,daily,weekly,monthly}</code> and {{C|/etc/boot.d}}.


    === Apache modules, config, and sites ===
    === Apache modules and configuration ===


    Enable PHP FPM:
    Enable PHP FPM and other Apache modules:


      a2enmod proxy_fcgi
      a2enmod expires headers proxy_fcgi rewrite
      a2enconf php${php_version}-fpm
      a2enconf php${php_version}-fpm


    We need a number of Apache modules to be enabled which might not be enabled by default:
    === OpenLDAP configuration database ===
     
    First of all, check if there are important changes in the base configuration of slapd.  For this, we can copy the old configuration into some directory on the new server and run a recursive diff:
     
    # Run on old server
    rsync -az /etc/ldap/slapd.d/ feministwiki.dev:/tmp/slapd.d
     
    # Run on new server
    diff -ru --color=always /tmp/slapd.d /etc/ldap/slapd.d | less -R
     
    There are going to be a number of changes that are expected.  Namely:
     
    # CRCs, UIDs, timestamps, and other such auto-generated fields
    # FeministWiki-specific things that only exists in the old configuration
     
    If these are the '''only''' differences you can see, then it should be safety to completely override the config on the new server with the old one, using the instructions in the following section.


    a2enmod expires headers macro rewrite ssl
    Otherwise, skip to the section after that and recreate the FeministWiki-specific configuration from scratch.
    a2enconf 99-local
    a2ensite fw-account fw-blogs fw-chat fw-files fw-forum fw-mail fw-wiki fw-xmpp


    === OpenLDAP configuration database ===
    ==== Complete copying of old configuration ====
     
    ''Note: This is an '''alternative''' method to that described in the '''next''' section. See above for which one to choose.''


    Stop the LDAP server and delete the configuration database '''on the new server (careful!)''':
    Stop the LDAP server and delete the configuration database '''on the new server (careful!)''':
    Line 212: Line 269:
    Then copy over the configuration database, by running the following commands from the old server:
    Then copy over the configuration database, by running the following commands from the old server:


    # Run on old server
      slapcat -n 0 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d'
      slapcat -n 0 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d'
    ==== Recreation of FeministWiki configuration ====
    ''Note: This is an '''alternative''' method to that described in the '''previous''' section. See above for which one to choose.''
    First run {{C|dpkg-reconfigure slapd}} to fill in some basic information such as the domain name and admin password.  You can reuse the old admin password found in {{C|/root/pwd/ldap}}.
    Then, running the following sequence of commands, taken from [[FeministWiki:LDAP Schema]], should do the rest:
    # Create fwMember object class
    ldapadd -Y external -H ldapi:// <<EOF
    dn: cn=feministwiki,cn=schema,cn=config
    objectClass: olcSchemaConfig
    cn: feministwiki
    olcAttributeTypes: {0}( 1.3.6.1.4.1.42.2.27.99.1.1
        NAME 'fwRecoveryMail'
        DESC 'FeministWiki password recovery mail'
        EQUALITY caseIgnoreMatch
        SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )
    olcObjectClasses: {1}( 1.3.6.1.4.1.42.2.27.99.2.1
        NAME 'fwMember'
        DESC 'FeministWiki member'
        SUP inetOrgPerson
        STRUCTURAL
        MAY ( fwRecoveryMail ) )
    EOF
    # Set attribute permissions
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: olcDatabase={1}mdb,cn=config
    changetype: modify
    add: olcAccess
    olcAccess: {2}to attrs=sn,mail by self write
    olcAccess: {3}to attrs=fwRecoveryMail by self write by dn.exact="cn=readonly,dc=feministwiki,dc=org" search
    olcAccess: {4}to attrs=manager by self read
    EOF
    # Enable the ppolicy dynamic module
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: cn=module{0},cn=config
    changetype: modify
    add: olcModuleLoad
    olcModuleLoad: ppolicy
    EOF
    # Add the ppolicy overlay with olcPPolicyHashCleartext set to TRUE
    ldapadd -Y external -H ldapi:// <<EOF
    dn: olcOverlay=ppolicy,olcDatabase={1}mdb,cn=config
    objectClass: olcPPolicyConfig
    olcOverlay: ppolicy
    olcPPolicyHashCleartext: TRUE
    EOF
    # Set the default password policy
    # The policy object referenced here doesn't exist yet,
    # but will exist once we copy over the main database.
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: olcOverlay={0}ppolicy,olcDatabase={1}mdb,cn=config
    changetype: modify
    add: olcPPolicyDefault
    olcPPolicyDefault: cn=default,ou=pwdPolicies,dc=feministwiki,dc=org
    EOF
    # Load the lastbind module
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: cn=module{0},cn=config
    changetype: modify
    add: olcModuleLoad
    olcModuleLoad: lastbind
    EOF
    # Enable the lastbind overlay
    ldapadd -Y external -H ldapi:// <<EOF
    dn: olcOverlay=lastbind,olcDatabase={1}mdb,cn=config
    objectClass: olcLastBindConfig
    olcOverlay: lastbind
    olcLastBindPrecision: 60
    EOF


    ==== Breaking changes in OpenLDAP ====
    ==== Breaking changes in OpenLDAP ====
    Line 257: Line 393:
    === LDAP database ===
    === LDAP database ===


    Delete the existing database '''on the new server (careful!)''':
    Make sure slapd is not running and delete the existing database '''on the new server (careful!)''':


    # Run on new server!
    systemctl stop slapd
      rm /var/lib/ldap/data.mdb
      rm /var/lib/ldap/data.mdb


    Then copy over the database by running the following command from the old server:
    Then copy over the database by running the following command from the old server:


      slapcat -n 1 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 1'
    # Run on old server
      slapcat -n 1 | zstd | ssh feministwiki.dev 'zstd -d | sudo -u openldap slapadd -n 1'


    Although there may be breaking changes that make this command fail, just as with the copying of the configuration database explained earlier, the chance is much lower for the regular "data" database, so hopefully the command will run fine.
    Start slapd again in the new server afterwards:
     
    # Run on new server
    systemctl start slapd


    === Contents of /var/www ===
    === Contents of /var/www ===
    Line 271: Line 413:
    This is very simple but takes a lot of time to finish.  '''Run it from the old server:'''
    This is very simple but takes a lot of time to finish.  '''Run it from the old server:'''


      rsync -az --delete /var/www/ feministwiki.dev:/var/www
      rsync -azP --delete /var/www/ feministwiki.dev:/var/www


    Note that the trailing slash in {{C|/var/www/}} is important; if not provided, it will copy the directory to {{C|/var/www/www}} on the new server.
    Note that the trailing slash in {{C|/var/www/}} is important; if not provided, it will copy the directory to {{C|/var/www/www}} on the new server.
    There's actually a systemd service found in {{C|/var/www/fw/wiki}} that you'll want to enable on the new server:
    systemctl enable /var/www/fw/wiki/fw-wiki-job-runner.service
    No need to actually start it yet.


    === SQL databases ===
    === SQL databases ===
    Line 279: Line 427:
    Run the following command from the old server:
    Run the following command from the old server:


      mysqldump -u root -p"$(cat /root/pwd/mysql)" \
      mariadb-dump -u root -p"$(cat /root/pwd/mariadb)" \
       --add-drop-database \
       --add-drop-database \
       --databases blogs \
       --databases feministblogs \
                   feministfiles \
                   feministfiles \
                   feministforum \
                   feministforum \
    Line 292: Line 440:
                   feministwiki_pt \
                   feministwiki_pt \
                   fff \
                   fff \
       | gzip | ssh root@feministwiki.dev 'gunzip | /root/bin/sql'
       | zstd | ssh feministwiki.dev 'zstd -d | /root/bin/sql'


    You can use the {{C|show databases;}} command in the SQL console to make sure that the list of databases is complete.  Unfortunately they have to be listed manually, because using the {{C|--all-databases}} option includes system databases that we don't want to copy.
    You can use the {{C|show databases;}} command in the SQL console to make sure that the list of databases is complete.  Unfortunately they have to be listed manually, because using the {{C|--all-databases}} option includes system databases that we don't want to copy.
    Line 303: Line 451:


    Note that the trailing slash in {{C|/home/vmail/}} is important.
    Note that the trailing slash in {{C|/home/vmail/}} is important.
    === Elasticsearch ===
    Temporarily stop Elasticsearch on the old server and copy over the data:
    systemctl stop elasticsearch
    rsync -az --delete /var/lib/elasticsearch/ feministwiki.dev:/var/lib/elasticsearch
    systemctl start elasticsearch


    === Mailman data ===
    === Mailman data ===
    Line 322: Line 478:


      /root/bin/sql << EOF
      /root/bin/sql << EOF
      create user blogs@localhost identified by '$(cat ~/pwd/mysql-blogs)';
      create user 'feministblogs'@localhost identified by '$(cat ~/pwd/mariadb-feministblogs)';
      create user feministfiles@localhost identified by '$(cat ~/pwd/mysql-files)';
      create user 'feministfiles'@localhost identified by '$(cat ~/pwd/mariadb-feministfiles)';
      create user feministforum@localhost identified by '$(cat ~/pwd/mysql-forum)';
      create user 'feministforum'@localhost identified by '$(cat ~/pwd/mariadb-feministforum)';
      create user feministmail@localhost identified by '$(cat ~/pwd/mysql-mail)';
      create user 'feministmail'@localhost identified by '$(cat ~/pwd/mariadb-feministmail)';
      create user feministwiki@localhost identified by '$(cat ~/pwd/mysql-wiki)';
      create user 'feministwiki'@localhost identified by '$(cat ~/pwd/mariadb-feministwiki)';
      create user fff@localhost identified by '$(cat ~/pwd/mysql-fff)';
      create user 'fff'@localhost identified by '$(cat ~/pwd/mariadb-fff)';
      EOF
      EOF


    Line 333: Line 489:


      /root/bin/sql << EOF
      /root/bin/sql << EOF
      grant all on blogs.* to blogs@localhost;
      grant all on feministblogs.* to feministblogs@localhost;
      grant all on feministfiles.* to feministfiles@localhost;
      grant all on feministfiles.* to feministfiles@localhost;
      grant all on feministforum.* to feministforum@localhost;
      grant all on feministforum.* to feministforum@localhost;
    Line 383: Line 539:
      systemctl stop dovecot
      systemctl stop dovecot
      systemctl stop ejabberd
      systemctl stop ejabberd
    systemctl stop elasticsearch
    systemctl stop fw-wiki-job-runner
      systemctl stop inspircd
      systemctl stop inspircd
      systemctl stop mailman
      systemctl stop nginx
    systemctl stop opendkim
      systemctl stop postfix
      systemctl stop postfix
      systemctl stop slapd
      systemctl stop slapd
    Note that we leave MariaDB running, since it needs to be live for data transfer.


    == Finishing up ==
    == Finishing up ==
    Line 396: Line 557:
    Stop all the services that interface with users and/or are responsible for modifying live data:
    Stop all the services that interface with users and/or are responsible for modifying live data:


    for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
    do ufw delete allow proto tcp to 0.0.0.0/0 port $port
    done
      systemctl stop apache2
      systemctl stop apache2
      systemctl stop dovecot
      systemctl stop dovecot
      systemctl stop ejabberd
      systemctl stop ejabberd
    systemctl stop elasticsearch
    systemctl stop fw-wiki-job-runner
      systemctl stop inspircd
      systemctl stop inspircd
      systemctl stop mailman
      systemctl stop nginx
    systemctl stop opendkim
      systemctl stop postfix
      systemctl stop postfix
      systemctl stop slapd
      systemctl stop slapd


    Close all the relevant ports just to be double-sure:
    As with the old server, we leave MariaDB running since it will be needed for data transfer.
     
    for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
    do ufw delete allow proto tcp to 0.0.0.0/0 port $port
    done


    === Copy over the live data one more time ===
    === Copy over the live data one more time ===
    Line 414: Line 578:
    '''Simply repeat the whole section ''Copying over live data''.'''
    '''Simply repeat the whole section ''Copying over live data''.'''


    The techniques and commands described above in the section ''Copying over live data'' are ''idempotent'', meaning you can simply repeat them and they will make sure that the new copy of the live data is fresh and doesn't leave any outdated data on the new server.  For instance, the {{C|--delete}} argument to the {{C|rsync}} command and the {{C|--add-drop-database}} argument to the {{C|mysqldump}} command help to make sure of this.
    The techniques and commands described above in the section ''Copying over live data'' are ''idempotent'', meaning you can simply repeat them and they will make sure that the new copy of the live data is fresh and doesn't leave any outdated data on the new server.  For instance, the {{C|--delete}} argument to the {{C|rsync}} command and the {{C|--add-drop-database}} argument to the {{C|mariadb-dump}} command help to make sure of this.


    So just repeat the steps from that section exactly one more time.
    So just repeat the steps from that section exactly one more time.

    Latest revision as of 17:37, 1 June 2024

    These are the steps required to set up a new FeministWiki Debian or Ubuntu server. The guide assumes that you're comfortable connecting to a host with SSH and using the shell for various administration tasks, such as using systemctl, editing configuration files, installing packages, setting up SSH keys, and so on.

    Initial setup of the new server

    This section describes various initialization tasks for the new server that are independent of the old server.

    Configure reverse DNS

    In the settings of the VPS host (e.g. Strato AG), you can configure reverse-DNS for the IP address of the server. Set the FQDN for the IP address to feministwiki.org. It's good to do this early since it can take some time to propagate.

    Make feministwiki.dev point to the new server

    During setup and testing of the new server, we want to make it accessible under the feministwiki.dev domain. So change A entries of the feministwiki.dev DNS settings to point to the IP address of the new server.

    Update & upgrade

    First of all, let's make sure the system is up to date.

    apt-get update
    apt-get upgrade
    apt-get dist-upgrade
    

    Install miscellaneous tools

    Some of these are needed further down, some are just good to have.

    apt-get install automysqlbackup \
                    bsdutils \
                    certbot \
                    composer \
                    curl \
                    dnsutils \
                    emacs-nox \
                    git \
                    imagemagick \
                    iotop \
                    ldap-utils \
                    mg \
                    moreutils \
                    net-tools \
                    netcat-openbsd \
                    nmap \
                    rsync \
                    tree
    

    Copy SSH key from old server

    Copy over ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub from the old server onto the new one. Make sure to set the permissions for the private key correctly: chmod 600 ~/.ssh/id_rsa

    Set up SSH access from the old server

    Some of the commands further below need SSH access from the old server to the new server, using the feministwiki.dev domain name. Since we've just copied over the public key of the old server, we can enable access from the old server with this simple command:

    # Run on new server
    cat .ssh/id_rsa.pub >> .ssh/authorized_keys
    

    Further, to make our life easier, we can edit the SSH configuration on the old server so we don't have to manually specify the custom SSH port number every time. Add a block like the following into ~/.ssh/config on the old server, replacing <SSH_PORT> with the actual port number:

    Host feministwiki.dev
        Port <SSH_PORT>
    

    Tighten security of SSH access

    Port 22 will get lots of malicious login attempts. It's a good idea to change the SSH port, and also to disable password authentication in favor of key-based authentication. Both can be done by editing /etc/ssh/sshd_config.

    Before restarting the SSH service, make sure you've actually added your public key (the contents of ~/.ssh/id_rsa.pub on your computer) to /root/.ssh/authorized_keys on the server, or you'll lock yourself out.

    Set up firewall

    For now, block everything but SSH.

    apt-get install ufw
    ufw allow proto tcp to 0.0.0.0/0 port ${SSH_PORT} # Replace with actual port number
    ufw enable
    

    Fetch scripts & config repo

    Having copied the .ssh/id_rsa from the old server will give you access to the GitHub FeministWiki repo:

    cd
    mkdir repo
    git clone git@github.com:FeministWiki/FeministWiki.git repo/fw
    cp -ai repo/fw/root/* repo/fw/root/.??* .
    sh repo/fw/decrypt-pwd.sh
    

    The decryption script will prompt you for a password the first time it's used. Enter the password stored in /root/pwd/meta on the old server.

    Enable extra repositories

    We might want to add some additional package repositories so we can use the latest version of some of the used software.

    Backports is always OK to add since the packages don't get priority over the stable ones:

    # Debian
    # May not be necessary; see if there's a commented out line in /etc/apt/sources.list that you can activate.
    echo deb http://deb.debian.org/debian $(lsb_release -sc)-backports main contrib non-free > /etc/apt/sources.list.d/backports.list
    
    # Ubuntu
    echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-backports main universe > /etc/apt/sources.list.d/backports.list
    

    Usually you want up-to-date versions of Apache2, Nginx, and PHP. Ondřej provides them:

    # Debian
    for repo in apache2 nginx-mainline php
    do
      curl https://packages.sury.org/$repo/apt.gpg > /etc/apt/trusted.gpg.d/sury-$repo.gpg
      echo "deb https://packages.sury.org/$repo/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/sury-$repo.list
    done
    
    # Ubuntu
    add-apt-repository ppa:ondrej/apache2
    add-apt-repository ppa:ondrej/nginx
    add-apt-repository ppa:ondrej/php
    

    It may be necessary to increase the priority of some repositories if you encounter errors about dependencies that can't be fulfilled, due to apt-get thinking it has to prioritize older versions of some packages. This was last observed while trying to install nginx-extras, due to some library dependencies having the "epoch" part of the version number set in the regular Debian repository, but not set in the Sury repository. The solution is to create a file in /etc/apt/preferences.d such as:

    Package: *
    Pin: origin packages.sury.org
    Pin-Priority: 700
    

    Elasticsearch, if you want CirrusSearch for MediaWiki:

    curl https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /etc/apt/trusted.gpg.d/elasticsearch.gpg
    # As of January 2024, CirrusSearch only supports Elasticsearch 7.x
    echo 'deb https://artifacts.elastic.co/packages/7.x/apt stable main' > /etc/apt/sources.list.d/elastic.list
    

    Create vmail user

    groupadd -g 5000 vmail
    useradd -u 5000 -g vmail -s /usr/sbin/nologin -d /home/vmail -m vmail
    

    Install server components

    Now we can install all the software used for the various FeministWiki services:

    apt-get install
        apache2 \
        dovecot-core \
        dovecot-imapd \
        dovecot-ldap \
        dovecot-pop3d \
        ejabberd \
        elasticsearch \
        fail2ban \
        inspircd \
        mariadb-server \
        nginx-extras \
        opendkim \
        postfix \
        postfix-ldap \
        slapd
    

    If any installation asks you for a password, remember that most passwords are found in /root/pwd.

    Example for installing ejabberd from backports instead:

    apt-get install ejabberd/$(lsb_release -sc)-backports # e.g. ejabberd/bookworm-backports
    

    Make sure Postfix can connect to OpenDKIM

    mkdir -p /var/spool/postfix/opendkim
    chown opendkim:opendkim /var/spool/postfix/opendkim
    adduser postfix opendkim
    

    Install PHP and modules

    This should really be part of the last section, but due to the sheer number of PHP modules we want to install, it's in its own section:

    php_version=8.1 # or whatever version we're on
    
    apt-get install php${php_version} \
                    php${php_version}-apcu \
                    php${php_version}-bcmath \
                    php${php_version}-bz2 \
                    php${php_version}-cli \
                    php${php_version}-curl \
                    php${php_version}-fpm \
                    php${php_version}-gd \
                    php${php_version}-gmp \
                    php${php_version}-imagick \
                    php${php_version}-intl \
                    php${php_version}-ldap \
                    php${php_version}-mbstring \
                    php${php_version}-mysql \
                    php${php_version}-opcache \
                    php${php_version}-readline \
                    php${php_version}-xml \
                    php${php_version}-zip
    

    We also want php-luasandbox, which may not have a PHP version attached to the package name, in which case you'll have to make sure it supports the PHP version currently in use. If not, you can use the standalone Lua binary instead by setting $wgScribuntoDefaultEngine = 'luastandalone'; in MediaWiki's LocalSettings.php configuration file.

    # See if this works first:
    apt-get install php${php_version}-luasandbox
    
    # Otherwise...
    apt-get install php-luasandbox
    # Check the package contents to see which PHP versions are supported
    dpkg -L php-luasandbox
    

    Copy over certificates

    Copy over the certs from the old server:

    # Run on old server
    rsync -avz /etc/feministwiki/certs feministwiki.dev:/etc/feministwiki/certs
    

    The /etc/feministwiki/certs directory and its contents should be owned by the group ssl-cert. Make sure this is the case on the new server after running the command above, since the group ID might be different on the new server. If the group doesn't exist at all, just create it.

    Further, files in that directory which contain the private key (privkey.pem and bundle.pem) should only be readable by group members. That is, their permission mode should be 640, displayed as -rw-r----- in the output of ls -l. Make sure this really the case.

    Then, to allow certain services to read those files containing the private key, add them to the ssl-cert group:

    # Run on new server
    adduser ejabberd ssl-cert
    adduser irc ssl-cert
    

    Also copy over the certificates stored directly in /etc/letsencrypt:

    # Run on old server
    rsync -avz /etc/letsencrypt/{archive,live} feministwiki.dev:/etc/letsencrypt
    

    Put config files in place

    The principle is simple: take all the config files from /root/repo/etc and put them where they belong in /etc. However, since a new server might mean much newer software, it's possible that some config files aren't compatible anymore, or that some new sensible defaults might be overwritten by the old config. Sadly, figuring out these incompatibilities is a manual process: compare the new default config with the old default config, and/or with our own config saved in the repo, to figure out what our new config files should look like.

    There's a number of things important to remember:

    • Don't forget to revert the redactions of sensitive information. Search files for [REDACTED].
    • After copying /etc/aliases, run newaliases to create the alias database file.
    • After copying Postfix configuration, run postmap /etc/postfix/sender-access and postalias /etc/postfix/virtual-aliases.
    • Make sure the executable bit is set on all cron scripts, /etc/rc.local, and scripts in /etc/letsencrypt/renewal-hooks.

    Apache modules and configuration

    Enable PHP FPM and other Apache modules:

    a2enmod expires headers proxy_fcgi rewrite
    a2enconf php${php_version}-fpm
    

    OpenLDAP configuration database

    First of all, check if there are important changes in the base configuration of slapd. For this, we can copy the old configuration into some directory on the new server and run a recursive diff:

    # Run on old server
    rsync -az /etc/ldap/slapd.d/ feministwiki.dev:/tmp/slapd.d
    
    # Run on new server
    diff -ru --color=always /tmp/slapd.d /etc/ldap/slapd.d | less -R
    

    There are going to be a number of changes that are expected. Namely:

    1. CRCs, UIDs, timestamps, and other such auto-generated fields
    2. FeministWiki-specific things that only exists in the old configuration

    If these are the only differences you can see, then it should be safety to completely override the config on the new server with the old one, using the instructions in the following section.

    Otherwise, skip to the section after that and recreate the FeministWiki-specific configuration from scratch.

    Complete copying of old configuration

    Note: This is an alternative method to that described in the next section. See above for which one to choose.

    Stop the LDAP server and delete the configuration database on the new server (careful!):

    # Commands to run on the NEW (fresh) server:
    systemctl stop slapd
    rm -r /etc/ldap/slapd.d/*
    

    Then copy over the configuration database, by running the following commands from the old server:

    # Run on old server
    slapcat -n 0 | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d'
    

    Recreation of FeministWiki configuration

    Note: This is an alternative method to that described in the previous section. See above for which one to choose.

    First run dpkg-reconfigure slapd to fill in some basic information such as the domain name and admin password. You can reuse the old admin password found in /root/pwd/ldap.

    Then, running the following sequence of commands, taken from FeministWiki:LDAP Schema, should do the rest:

    # Create fwMember object class
    ldapadd -Y external -H ldapi:// <<EOF
    dn: cn=feministwiki,cn=schema,cn=config
    objectClass: olcSchemaConfig
    cn: feministwiki
    olcAttributeTypes: {0}( 1.3.6.1.4.1.42.2.27.99.1.1
       NAME 'fwRecoveryMail'
       DESC 'FeministWiki password recovery mail'
       EQUALITY caseIgnoreMatch
       SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )
    olcObjectClasses: {1}( 1.3.6.1.4.1.42.2.27.99.2.1
       NAME 'fwMember'
       DESC 'FeministWiki member'
       SUP inetOrgPerson
       STRUCTURAL
       MAY ( fwRecoveryMail ) )
    EOF
    
    # Set attribute permissions
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: olcDatabase={1}mdb,cn=config
    changetype: modify
    add: olcAccess
    olcAccess: {2}to attrs=sn,mail by self write
    olcAccess: {3}to attrs=fwRecoveryMail by self write by dn.exact="cn=readonly,dc=feministwiki,dc=org" search
    olcAccess: {4}to attrs=manager by self read
    EOF
    
    # Enable the ppolicy dynamic module
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: cn=module{0},cn=config
    changetype: modify
    add: olcModuleLoad
    olcModuleLoad: ppolicy
    EOF
    
    # Add the ppolicy overlay with olcPPolicyHashCleartext set to TRUE
    ldapadd -Y external -H ldapi:// <<EOF
    dn: olcOverlay=ppolicy,olcDatabase={1}mdb,cn=config
    objectClass: olcPPolicyConfig
    olcOverlay: ppolicy
    olcPPolicyHashCleartext: TRUE
    EOF
    
    # Set the default password policy
    # The policy object referenced here doesn't exist yet,
    # but will exist once we copy over the main database.
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: olcOverlay={0}ppolicy,olcDatabase={1}mdb,cn=config
    changetype: modify
    add: olcPPolicyDefault
    olcPPolicyDefault: cn=default,ou=pwdPolicies,dc=feministwiki,dc=org
    EOF
    
    # Load the lastbind module
    ldapmodify -Y external -H ldapi:// <<EOF
    dn: cn=module{0},cn=config
    changetype: modify
    add: olcModuleLoad
    olcModuleLoad: lastbind
    EOF
    
    # Enable the lastbind overlay
    ldapadd -Y external -H ldapi:// <<EOF
    dn: olcOverlay=lastbind,olcDatabase={1}mdb,cn=config
    objectClass: olcLastBindConfig
    olcOverlay: lastbind
    olcLastBindPrecision: 60
    EOF
    

    Breaking changes in OpenLDAP

    There might be incompatible changes between OpenLDAP (aka slapd) versions which require manual editing of the slapcat output before it's read in on the new server with slapadd. Following are two examples of this.

    These particular issues won't apply anymore when you're reading this guide, since they are one-time issues related to migrating to a newer OpenLDAP version, but they serve as good examples. (Also, no such clear explanation of the first problem seems to be found anywhere on the web, so maybe someone who searches the related error message below will come upon this guide and be happy!)

    OpenLDAP Migration: Example Problem 1

    This problem occurs when migrating from OpenLDAP 2.4.42 or earlier, to 2.4.43 or later.

    The ppolicy overlay gained a new attribute in OpenLDAP version 2.4.43, so if you simply run the command above which copies over the configuration database onto the new server, it will produce the following error message:

    User Schema load failed for attribute "pwdMaxRecordedFailure". Error code 17: attribute type undefined
    

    The solution is as follows:

    1. On the new server, open /etc/ldap/schema/ppolicy.ldif and search for pwdMaxRecordedFailure. You will see an olcAttributeTypes: ... entry that defines it. Also, it's listed in the MAY attributes block of the olcObjectClasses: ... entry that defines the pwdPolicy object class.
    2. On the old server, save the output of slapcat -n 0 to a file, open it in a text editor, and search for the block where the ppolicy schema is defined. It should start with the line dn: cn={4}ppolicy,cn=schema,cn=config (the {4} part might contain a different integer, that's OK). There, note that the olcAttributeTypes: ... entry for pwdMaxRecordedFailure is missing, and also it's not listed in the MAY list of the pwdPolicy object class definition. Copy over the attribute type definition from the ppolicy.ldif file on the new server, and amend the MAY list to include it.
    OpenLDAP Migration: Example Problem 2

    This problem occurs when migrating to OpenLDAP 2.5, and although the change is bigger, the fix is easier. The issue is actually documented in the OpenLDAP 2.5 Administrator's Guide, Appendix B.2:

    https://www.openldap.org/doc/admin25/appendix-upgrading.html

    The second paragraph tells us what to do:

    In OpenLDAP 2.4 the slapo-ppolicy(5) overlay relied on a separate schema file to be included for it to function. This schema is now implemented internally in the slapo-ppolicy module. When upgrading slapd.conf(5) deployments the include statement for the schema must be removed. For slapd-config(5) deployments, the config database must be exported via slapcat and the old ppolicy schema removed from the export. The resulting config database can then be imported.

    In simpler terms:

    1. Save the output of slapcat -n 0 from the old server in a file:
      slapcat -n 0 > slapcat.n0.out
    2. Open the file in a text editor and delete the block starting with the line dn: cn={4}ppolicy,cn=schema,cn=config, up to the next empty line (before the next block starting with a dn: ... line), and save the file.
    3. Feed the file to slapadd -n 1 on the new server:
      cat slapcat.n0.out | ssh feministwiki.dev 'sudo -u openldap slapadd -n 0 -F /etc/ldap/slapd.d'

    Copying over live data

    We want to make a first run of this copy process purely for testing purposes. Note that although some of the steps described in this section take a long time to finish, they can be done in parallel.

    LDAP database

    Make sure slapd is not running and delete the existing database on the new server (careful!):

    # Run on new server!
    systemctl stop slapd
    rm /var/lib/ldap/data.mdb
    

    Then copy over the database by running the following command from the old server:

    # Run on old server
    slapcat -n 1 | zstd | ssh feministwiki.dev 'zstd -d | sudo -u openldap slapadd -n 1'
    

    Start slapd again in the new server afterwards:

    # Run on new server
    systemctl start slapd
    

    Contents of /var/www

    This is very simple but takes a lot of time to finish. Run it from the old server:

    rsync -azP --delete /var/www/ feministwiki.dev:/var/www
    

    Note that the trailing slash in /var/www/ is important; if not provided, it will copy the directory to /var/www/www on the new server.

    There's actually a systemd service found in /var/www/fw/wiki that you'll want to enable on the new server:

    systemctl enable /var/www/fw/wiki/fw-wiki-job-runner.service
    

    No need to actually start it yet.

    SQL databases

    Run the following command from the old server:

    mariadb-dump -u root -p"$(cat /root/pwd/mariadb)" \
      --add-drop-database \
      --databases feministblogs \
                  feministfiles \
                  feministforum \
                  feministmail \
                  feministwiki \
                  feministwiki_de \
                  feministwiki_es \
                  feministwiki_fr \
                  feministwiki_it \
                  feministwiki_pt \
                  fff \
      | zstd | ssh feministwiki.dev 'zstd -d | /root/bin/sql'
    

    You can use the show databases; command in the SQL console to make sure that the list of databases is complete. Unfortunately they have to be listed manually, because using the --all-databases option includes system databases that we don't want to copy.

    Emails

    This is a simple one. Run this command from the old server:

    rsync -az --delete /home/vmail/ feministwiki.dev:/home/vmail
    

    Note that the trailing slash in /home/vmail/ is important.

    Elasticsearch

    Temporarily stop Elasticsearch on the old server and copy over the data:

    systemctl stop elasticsearch
    rsync -az --delete /var/lib/elasticsearch/ feministwiki.dev:/var/lib/elasticsearch
    systemctl start elasticsearch
    

    Mailman data

    GNU Mailman uses a filesystem-based "database" so we can transfer over its data as follows; run this from the old server:

    cd /var/lib/mailman
    rsync -az --delete archives data lists feministwiki.dev:/var/lib/mailman
    

    And then this on the new server:

    check_perms -f
    

    The check_perms command, which is part of GNU Mailman, will take care of fixing file ownership and permissions.

    Recreate SQL users

    If the versions of MariaDB on the old and new server are compatible enough, you might be able to dump the mysql.user table and import it on the new server, but it's safer to recreate the users from scratch. To do so, run this on the new server:

    /root/bin/sql << EOF
    create user 'feministblogs'@localhost identified by '$(cat ~/pwd/mariadb-feministblogs)';
    create user 'feministfiles'@localhost identified by '$(cat ~/pwd/mariadb-feministfiles)';
    create user 'feministforum'@localhost identified by '$(cat ~/pwd/mariadb-feministforum)';
    create user 'feministmail'@localhost identified by '$(cat ~/pwd/mariadb-feministmail)';
    create user 'feministwiki'@localhost identified by '$(cat ~/pwd/mariadb-feministwiki)';
    create user 'fff'@localhost identified by '$(cat ~/pwd/mariadb-fff)';
    EOF
    

    Now grant them access to their corresponding databases. Remember that this has to be done after the databases have been copied over:

    /root/bin/sql << EOF
    grant all on feministblogs.* to feministblogs@localhost;
    grant all on feministfiles.* to feministfiles@localhost;
    grant all on feministforum.* to feministforum@localhost;
    grant all on feministmail.* to feministmail@localhost;
    grant all on feministwiki.* to feministwiki@localhost;
    grant all on feministwiki_de.* to feministwiki@localhost;
    grant all on feministwiki_es.* to feministwiki@localhost;
    grant all on feministwiki_fr.* to feministwiki@localhost;
    grant all on feministwiki_it.* to feministwiki@localhost;
    grant all on feministwiki_pt.* to feministwiki@localhost;
    grant all on fff.* to fff@localhost;
    EOF
    

    Test

    It's important to test the new server to make sure everything works well!

    Reboot

    We could restart a lot of services manually to ensure they've read their new config, but it's easiest to just reboot. (The new server, obviously.)

    Open ports

    We need to open all the ports used by the various FeministWiki services:

    for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
    do ufw allow proto tcp to 0.0.0.0/0 port $port
    done
    

    Test!

    At this point you should test everything using the feministwiki.dev domain name.

    Some things may not work correctly because they're hard-coded to work as "feministwiki.org" and not under the "feministwiki.dev" name. Here's a list of known issues related to this:

    • WordPress normally redirects clients to the canonical address of a blog if it's visited through an alternative domain name. This means we get redirected back to the old server if we try to visit blogs.feministwiki.dev. To work around this, we use RequestHeader set Host in the Apache2 site configuration, which fools WordPress into believing it's being accessed through the canonical domain name. Still, the HTML/CSS/JS sent back to the browser refers to some resources on the .org domain, which then fail to load due to CORS violation.

    If you want to be extra thorough, you can edit your /etc/hosts file to make feministwiki.org, various *.feministwiki.org subdomains, and maybe even other aliases (such as fem.wiki) point to the new server, and then test the few stubborn services that won't otherwise play nice.

    Deactivate again

    After we're done testing, we can "deactivate" the new server again to prepare it for the final switch-over:

    for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
    do ufw delete allow proto tcp to 0.0.0.0/0 port $port
    done
    
    systemctl stop apache2
    systemctl stop dovecot
    systemctl stop ejabberd
    systemctl stop elasticsearch
    systemctl stop fw-wiki-job-runner
    systemctl stop inspircd
    systemctl stop nginx
    systemctl stop opendkim
    systemctl stop postfix
    systemctl stop slapd
    

    Note that we leave MariaDB running, since it needs to be live for data transfer.

    Finishing up

    Now, all services on the old server should be stopped, because we will begin the final transfer of live data.

    Stop services on the old server

    Stop all the services that interface with users and/or are responsible for modifying live data:

    for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
    do ufw delete allow proto tcp to 0.0.0.0/0 port $port
    done
    
    systemctl stop apache2
    systemctl stop dovecot
    systemctl stop ejabberd
    systemctl stop elasticsearch
    systemctl stop fw-wiki-job-runner
    systemctl stop inspircd
    systemctl stop nginx
    systemctl stop opendkim
    systemctl stop postfix
    systemctl stop slapd
    

    As with the old server, we leave MariaDB running since it will be needed for data transfer.

    Copy over the live data one more time

    Simply repeat the whole section Copying over live data.

    The techniques and commands described above in the section Copying over live data are idempotent, meaning you can simply repeat them and they will make sure that the new copy of the live data is fresh and doesn't leave any outdated data on the new server. For instance, the --delete argument to the rsync command and the --add-drop-database argument to the mariadb-dump command help to make sure of this.

    So just repeat the steps from that section exactly one more time.

    Reboot the new server

    At this point we can reboot the new server again, to make sure all services are properly restarted.

    Open ports on the new server

    Now we can open the ports again on the new server:

    for port in 25 80 443 465 587 993 995 5222 5223 5269 5270 5443 6697 7777
    do ufw allow proto tcp to 0.0.0.0/0 port $port
    done
    

    Update DNS entries

    You have to change the configuration of the following domains:

    • feministwiki.org
    • feministwiki.net
    • feministwiki.de
    • fem.wiki
    • feminism.wiki
    • feminist.wiki
    • fffrauen.de

    feministwiki.org

    You only have to change three DNS entries, since most of the subdomains work via CNAME entries:

    • The main A entry for @ (self-reference i.e. feministwiki.org)
    • The A entry for smtp since this is not allowed to be a CNAME
    • The A entry for xmpp since this is not allowed to be a CNAME

    Other domains

    For these, you only have to change the main A entry, since they don't use SMTP or XMPP.

    Configure LetsEncrypt

    Since we changed the DNS settings, we can now set up certbot:

    certbot register -n --agree-tos -m technician@feministwiki.org
    letsencrypt-refresh
    

    Finishing up

    At this point, everything should be functional. If not, it's time for some debugging!