Server guides for the GIRAF project

A joint guide produced by SW611 & SW616

Aalborg University

Contents

1 Accessing the GIRAF servers (SW611) . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Using Docker (SW611) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Add database user (SW616) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4 Links to all services (SW616) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5 Expand disk with LVM (SW616) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

6 Restore from backup (SW616) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

6.1 Jenkins 106.2 ApplicationDB 116.3 Phabricator 12

7 Service Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

7.1 Git (SW611) 137.2 Jenkins (SW616) 147.3 Artifactory (SW611) 157.4 Phabricator (SW616) 15

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1. Accessing the GIRAF servers (SW611)

There are four (4) servers accessible to the designated server group(s). This section will specifygeneral information with respect to these servers, and conventions used when managing theseservers. The services running on each server are specified in section 4.

web.giraf.cs.aau.dkThe web server can be accessed from outside the AAU internal network, through SSH usingport 22.

git.giraf.cs.aau.dkThe git server can be accessed from outside the AAU internal network, through SSH usingport 22.

jenkins.giraf.cs.aau.dkThe jenkins server can be accessed from outside the AAU internal network, through SSHusing port 22.

backup01.giraf.cs.aau.dkThe backup server cannot be accessed from outside the AAU internal network. If you wishto access the server from your private network, without using a VPN to AAU, you may SSHto the backup server from either of the other three (3) servers.

When first accessing the servers, we recommend that all created accounts be fitted with an SSH keyimmediately, to ensure good security. If you are not experienced with SSH keys, this is a fittingguide.

For accessing the servers, the SSH client PuTTY is recommended. PuTTY includes an SSH keygenerator, PuTTYgen, that may be used to generate the SSH keys.

We have disabled SSH login to root on all servers, and recommend that you do not change this. Ifyou wish to access root, use the sudo command.

When you want to access each of the servers for the first time, we have to setup a new accountfor you on all of them. Contact either of the following, for support with the initial user creationprocess:

mclc13@student.aau.dk - Mikkelsbjerg13@student.aau.dk - Sørensela12@student.aau.dk - Simon

2. Using Docker (SW611)

All servers, except for the backup server, uses the Docker platform for the services running on them.We recommend that you read up on the documentation for Docker Engine in order to understandhow Docker works. Further documentation specific to GIRAF, and argumentation for choosingDocker, may be found in the reports from groups SW611 and SW616.

3. Add database user (SW616)

One of the most requested ad hoc task is to add database users for programmers. This is done usingthe following commands.

• mysql -hweb.giraf.cs.aau.dk -P3333 -uroot -p (connects to the database server as root)• CREATE USER ’username’@’%’ IDENTIFIED BY ’password’ (create user with username

’username’ and password ’password’. User has access from all ip addresses (%));• GRANT SELECT ON giraf.* TO ’username’@’%’; (grant user access to select on all tables

in the database giraf)• GRANT SELECT,INSERT,DELETE,UPDATE,ALTER ON girafdev.* TO ’username’@’%’;

(grant user access to select, insert, delete, update and alter table in the database girafdev)• SHOW GRANTS FOR ’username’@’%’ (Show grants assigned to user);

4. Links to all services (SW616)

This chapter provides information about all running services and how to access these.

Jenkins

URL: http://jenkins.giraf.cs.aau.dkLogin using AAU Access Control

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

Phabricator

URL: http://web.giraf.cs.aau.dkLogin using AAU Access Control

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

Git

URL: http://giraf.giraf.cs.aau.dkLogin using credentials from AAU account

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

ApplicationDB

Address: web.giraf.cs.aau.dk:3333Login using credentials provided by server group

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

SymmetricDS

Address: web.giraf.cs.aau.dk:31415Login using credentials provided by server group

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes No

7

WebadminURL: http://web.giraf.cs.aau.dk:8081Login using credentials provided by server group

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

Wildfly PublicURL: http://web.giraf.cs.aau.dk:8080No login

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

Wildfly Administration ConsoleURL: http://web.giraf.cs.aau.dk:9990Login using credentials provided by server group

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes No

ArtifactoryURL: http://jenkins.giraf.cs.aau.dk:8080No login for normal users. Admin-login with credentials provided by server group

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

GlancesURLs: http://{jenkins,web,git}.giraf.cs.aau.dk:61208No login needed

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes No

FilestoreURL: http://jenkins.giraf.cs.aau.dk:8083/filesSFTP: jenkins.giraf.cs.aau.dk:2222No login needed

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes No

JavadocsURL: http://git.giraf.cs.aau.dk:8080No login needed

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes Yes

8 Chapter 4. Links to all services (SW616)

AdminerURL: http://git.giraf.cs.aau.dk:10080Login using credentials provided by server group

Accessible from servers Accessible inside AAU Accessible outside AAUYes Yes No

5. Expand disk with LVM (SW616)

To expand the volume with LVM the following commands has been used [1]:

• reboot (first reboot to make CentOS detect the new space)• fdisk -l (show disks - find which disk which needs to be added / has added space)• fdisk /dev/sdb (change partition table on /dev/sdb/)

Select n (new partition)Select p (primary)Select 1 (select next number available)Enter (First cylinder)Enter (Last cylinder)t (select partition)8e (Linux LVM filesystem)w (Write changes)

• fdisk -l (show disks - if new LVM is not presents - reboot)• pvcreate /dev/sda3 (create new volume for use with LVM)• vgdisplay (Show status and name for volume group which needs to be expanded)• vgextend centos /dev/sda3 (Extend volume group centos)• pvscan (Scan for new volume group)• lvdisplay (Show LVM status, also show name of LVM volume)• lvextend -L+2G /dev/centos/root /dev/sda3 (Extend volume /dev/centos/root with 2 GB from

/dev/sda3 (you can exclude -L+2G if you want to use all space))• xfs_growfs /dev/centos/root (Grow file system - detect new space)• df -h (Show status for file system)

6. Restore from backup (SW616)

Let’s assume that everything has crashed and you are forced to reinstall the servers. This guide willexplain how to get everything setup again if you are in this situation. Please note that all commandsare assumed to be run as root.

The first step is to install Docker on the server. Docker’s official installation guide is found here:https://docs.docker.com/engine/installation/linux/ubuntulinux

6.1 JenkinsNow that Docker is installed, we can create a container for Jenkins. We will be using the officialimage for this, this is installed with:

docker run --name jenkins -p 80:8080 -v /srv/jenkins_home:/var/jenkins_home jenkins

Jenkins should now be accessible at port 80, check that this is the case.

We now have a fresh, working Jenkins. In order to retrieve the old data, we need to do this from thebackup server. We need to stop the Jenkins container before this. This is done with:

docker stop jenkins

Now we delete all files in /srv/jenkins_home with command

cd /srv/jenkins_home; rm -rf *

Now we have to get into the backup-server, called backup01.giraf.cs.aau.dk and copy thebackup to the Jenkins server with the following commands:

cd /srv/backup/jenkins/jenkinsscp "jenkins.{DATE}-{TIME}.tar.gz" {USER}@{IP}:/srv/jenkins_home

Now we go back to the new server running Jenkins and extracts the backup with this command:

cd /srv/jenkins_hometar -zxvf jenkins.{DATE}-{TIME}.tar.gzcd /srv/jenkins_home/srv/backup/jenkins/jenkins/jenkins.{DATE}-{TIME}

6.2 ApplicationDB 11

mv * /srv/jenkins_home

All files are copied and placed in the correct directory, and we are ready to run

docker start jenkins

6.2 ApplicationDBFirst step is to install the MariaDB container with this command:

docker run --name applicationdb -v /srv/applicationdb:/var/lib/mysql-e MYSQL_ROOT_PASSWORD={PASSWORD} mariadb

When the database is running, we have to stop it before copying from the backup:

docker stop applicationdb

We can now copy the neccessary files from the backup with:

cd /srv/backup/web/applicationdbscp "applicationdb.{DATE}-{TIME}.tar.gz" {USER}@{IP}:/srv/application

Back on the new server we extract and move the neccessary files:

cd /srv/applicationdbtar -zxvf applicationdb.{DATE}-{TIME}.tar.gzcd /srv/applicationdb/srv/backup/web/applicationdb/applicationdb.{DATE}-{TIME}mv giraf /srv/applicationdbmv girafdev /srv/applicationdbmv girafRestdev /srv/applicationdb

Everything is ready for starting the container again with:

docker start applicationdb

12 Chapter 6. Restore from backup (SW616)

6.3 PhabricatorIn order to restore Phabricator, a little more work has to be done since this is split into to containers- one for the database, and one for the web server. Let us start by installing both of these:

docker run --name phabricator-database yesnault/docker-phabricator-mysqldocker run --name phabricator-web -p 80:80 --link phabricator-database:databaseyesnault/docker-phabricator

Since we need to modify data files, we need to stop them:

docker stop phabricator-database phabricator-web

We now have to clean up the generated database files by running:

cd $(docker inspect -f ’{{ (index .Mounts 0).Source }}’ phabricator-database);rm -rf default_* mysql performance_schema

We can now copy the neccessary files from the backup with:

cd /srv/backup/web/phabricator-databasescp "phabricator-database.{DATE}-{TIME}.tar.gz" {USER}@{IP}:$(docker inspect-f ’{{ (index .Mounts 0).Source }}’ phabricator-database)

Back on the new server we extract and move the neccessary files:

cd ~tar -zxvf phabricator -database .{DATE}-{TIME}.tar.gzcd ~/srv/backup/web/phabricator -database .{DATE}-{TIME}mv mysql $(docker inspect -f ’{{ (index .Mounts 0).Source }}’

phabricator -database)mv default_* $(docker inspect -f ’{{ (index .Mounts 0).Source }}’

phabricator -database)}mv phabricator_* $(docker inspect -f ’{{ (index .Mounts 0).Source

}}’ phabricator -database)}mv performance\_schema $(docker inspect -f ’{{ (index .Mounts

0).Source }}’ phabricator -database)}

The database server is ready to be started again, and then it’s time to repeat the above steps forphabricator-web

7. Service Administration

7.1 Git (SW611)Accessible at: http://git.giraf.cs.aau.dk

The system has been configured to only allow access to members of specific groups. Obviously thisis currently defined as those participating in the GIRAF multi–project in 2016. Specific rules aredefined for members of the two server groups, as well as for Ulrik, to ensure administrative rights.

To gain access to the system, the easiest way is to contact tbendi13@student.aau.dk, althoughany of the server group members should be able to assist.

User sign–up procedure

When a new user signs into GOGS for the first time, their account is created, but they aren’tautomatically added to the “Giraf” organization and its “Developers” team. To give users writeaccess to the repositories, an admin must add them to the team manually. It’s a bit of a hassle, but itdoesn’t take long, and the process is basically:

• Open the Giraf organization. The easiest way to do this is to find it in the list of organizationson the dashboard, in the right–hand side.

• Open the “Developers” team, which can be found in the right–hand side on the organizationpage. It defaults to the list of members.

• At the end of this list there’s a text field which allows you to add someone to the team. Prettysimple once you know about it.

New repositories, special teams, and access modifications

When adding a new repository it will automatically become available to the “Owners” team, but inorder to give developers access, it must be added to the “Developers” team. This is done in muchthe same way as adding a user to the team, only the “Repositories” link in the left–hand side infobar (on the team page) needs to be clicked first.

There are a couple of special teams as well. Here’s a quick rundown of all the teams, for goodmeasure:Owners:

Server group members as well as Ulrik should be added to this one. They should be able toadd themselves to it, though, so that shouldn’t be a problem. All repositories under the “Giraf”organization are automatically available with write access to all members of this group.

Developers:

14 Chapter 7. Service Administration

As the name suggests, this is where the “regular” members should be. It will give them fullaccess to all the repositories that have been added there.

DeprecatedRepos:There were a bunch of seemingly unused repositories on the old server, but rather thandeleting them, they have been added to this group. These are the repositories that werespecifically marked as deprecated and deletable, but for some reason they weren’t deleted, soit shall now be up to you to decide on whether or not they should stay or go.

UndecidedFate:These are more or less in the same category as the deprecated repositories, except that theformer server group didn’t mark them as deprecated specifically.

AdminSpecific:Hasn’t been maintained as well as it should be. The idea was that repositories that weremeant only for server admins’ eyes should be added here. Admins wouldn’t need to be addedto the group, so it’d mostly just be to get a quick overview of the admin repos.

DevRO:Not currently used, but if there’s ever a repository that the developers should have read–onlyaccess to, then that’s what this group is for. Obviously the developers in question would alsoneed to be added to the group, so there’s that.

JavaDocs:The documentation generation application uses API access to GOGS to find the repositoriesthat it needs to clone and generate for. So adding a repository to this team will enable it togenerate documentation for it, and I’m sure you can work out what removing a repo from theteam will do.

7.2 Jenkins (SW616)Accessible at: http://jenkins.giraf.cs.aau.dk

Jenkins is viewable and editable for everyone with a AAU login. However, administration is onlypossible for the server group. If you wish to gain administration access, please contact the servergroup from 2016.

There are a few important things that you need to know about. First of all, the location for key filesto Google Play is located at /srv/jenkins_home/google_play_keys. This directory containskey files as well as credentials for these.

Another important thing to keep in mind is that all configs for the project are saved via versioncontrol. This means that every single change to the config will be saved, and it’s possible to selectan older version.

A last remark about Jenkins is how this is restarted correctly. It’s important that the containerreceives the correct appends to the hosts file in order for Artifactory to work correctly. Thismeans that instead of running docker start jenkins, you need to use the bash script located at/home/mclc/restart_jenkins.sh

7.3 Artifactory (SW611) 15

7.3 Artifactory (SW611)Accessible at: http://jenkins.giraf.cs.aau.dk:8080

Artifactory is viewable by anyone able to access the URL. Artifactory administration can be donemanually using an admin login, credentials for which is stored in the/srv/jenkins_home/credentials/artifactory.properties file.

When logged in as admin, it is possible to import a system and repository to Artifactory, usingthe Admin panel, and this is where restoring the service from a backup can potentially be performed.

Artifactory has not needed any maintenance from either of our groups, ever since we migrated tothe current servers, and we do not expect this to change in the future.

7.4 Phabricator (SW616)Accessible at: http://web.giraf.cs.aau.dk/

Phabricator can be accessed using AAU login, which creates a new Phabricator user the first timeyou log in. The Phabricator account’s username is the first part of your AAU mail. We suggest thateach user also sets their real name, and include their group number, e.g. sela12 (Simon EllegaardLarsen (616)).

Regular users are permitted to do everything that is required for normal operation, so the adminis-trator account is not needed. However, there is a shared admin account in case a situation ariseswhere it is needed. The login credentials for this account can be found on the codesheet handed toUlrik (ulrik@cs.aau.dk).

Bibliography

[1] RootUsers. (May 10, 2016). How to increase the size of a linux lvm by adding a new disk,[Online]. Available: https://www.rootusers.com/how-to-increase-the-size-of-a-linux-lvm-by-adding-a-new-disk (cited on page 9).