The bioinformatics lab SS 2011

From Rost Lab Open
Jump to: navigation, search

This practical is a hands-on training that will make you successful in a Bioinformatic lab! This term we have a focus on virtual systems and Cloud Computing.

More details can be found at the http://rostlab.informatik.tu-muenchen.de/cms/biolab2011/ website.

Materials from old courses are available at The_bioinformatics_lab. There you can find many hints and tips and the protocols and presentation slides from last year. You may use them, but especially the protocols and presentations you have to prepare with your own content and own style!


Contents

External media / installing a Linux OS / Booting Linux / Debian stable

Programming challenge

Install and configure a Debian stable base system on your USB stick. Be careful with the partition, only use the USB device. Do not forget to enable boot on USB in your BIOS. Get familiar with vim.

Hints and tips

  1. Choose English so Laszlo can help
  2. Choose manual partitioning and make sure you leave alone all drives except the USB stick
  3. Note the device name of the USB stick, something like 'sda', 'sdb'...
  4. Have 3 partitions on the USB stick: one for a /boot file system which should be bootable and not more that 512 MB; one for swap space - 512MB; one for a / (root) file system. ext4 is a good choice for the /boot and / file systems.
  5. There is no proxy for accessing the internet
  6. Choose NOT to install the boot loader into the first hard drive: 'Install the GRUB boot loader to the master boot record?' NO
  7. Install the boot loader to the device you noted at step 3 above, e.g. /dev/sdb. Be careful not to install GRUB onto your hard drive.
  8. Boot from the USB key. Chances are this is going go fail because the way the installer installs GRUB (the boot loader) is not the best for removable devices.
  9. If booting fails with Error 17: Can not mount selected partition - the error we observed so far - hit 'e' on the grub screen to edit the first menu option. Hit 'e' again to edit the line root (hdX,Y). 'X' represents the hard drives as the BIOS sees them, numbered from 0. Error 17 indicates the the presently set hard drive does not contain a /boot partition where indicated by the 'root' line, so experiment with other numbers, e.g. 0: root (hd0,0). Do not change the second number 'Y': that indicates the partition and is likely to be correct. Try booting with the new 'root' line (hit 'b'). This modification is not permanent.
    1. Getting the 'root' line right should enable you to boot into Linux. The permanent solution is to use GRUB2 instead of GRUB. GRUB2 uses the universally unique identifier (UUID) to indicate root and boot partitions and so can handle removable devices well.
    2. Once you booted Linux log in as root and replace GRUB with GRUB2. You are going to need a working network connection for this step (courtesy of Fabian). Start the package manager 'aptitude'. Hit '?' and learn the keys' meanings. Search for the package 'grub2' and mark it for installation. Install it. Answer NO to the question whether to chain-load GRUB2. When the installation is complete exit aptitude.
    3. Install this version of grub to the master boot record of the USB key: execute update-grub. Find the current device name of the usb key, e.g. with mount, looking for a line like /dev/sda3 on /boot type ext4. In this case the USB stick is /dev/sda. Execute grub-install with the device for the USB key, e.g. grub-install /dev/sda.
    4. Reboot from the USB key - this time it should boot cleanly into Linux.
  10. Choose English as the default language of your system - use 'dpkg-reconfigure locales', add 'en_US.UTF-8' to the list of locales and make it the default
  11. You can change keyboard layout in the terminal with 'loadkeys'
  12. Install the 'vim-nox' package, call the 'vimtutor' command and start learning vim UNLESS you are proficient with emacs

Advanced challenge

Install Debian into an encrypted partition or encrypt the partition holding the home directories.

Report

By Verena Link:

Host names

Domain: course

User uid uidNumber Host name IP address VPN address i12r-studfilesrv user name
Laszlo lkajan 1002 lkajan.course 192.168.16.2 10.178.0.2 lkajan
Tatyana goldberg 1003 goldberg.course 192.168.16.3 10.178.0.3 goldberg
Erik pfeiff 1004 pfeiff.course 192.168.16.4 10.178.0.4 pfeiffenberger
Verena link 1005 link.course 192.168.16.5 10.178.0.5 link
Veit hoehn 1006 hoehn.course 192.168.16.6 10.178.0.6 hoehn
Antonia stank 1007 stank.course 192.168.16.7 10.178.0.7 stank
Evi bercht 1008 bercht.course 192.168.16.8 10.178.0.8 berchtold
Tanzeem charu 1009 charu.course 192.168.16.9 10.178.0.9 charu
Markus meier 1011 meier.course 192.168.16.11 10.178.0.11 meier
Petra nathan 1012 nathan.course 192.168.16.12 10.178.0.12 nathan
Max hecht 1013 hecht.course 192.168.16.13 10.178.0.13 hecht
Michael kiening 1014 kiening.course 192.168.16.14 10.178.0.14 kiening
Khadija elamrani 1015 elamrani.course 192.168.16.15 10.178.0.15 elamrani
Lothar richter 1016 richter.course 192.168.16.16 10.178.0.16 richter
Markus schmidb 1017 schmidb.course 192.168.16.17 10.178.0.17 n.a.
Esmeralda vicedo 1018 vicedo.course 192.168.16.18 10.178.0.18 vicedo
Stefan seemayer 1019 seemayer.course 192.168.16.19 10.178.0.19 seemayer
1020 .course 192.168.16.20 10.178.0.20
1021 .course 192.168.16.21 10.178.0.21

Shell scripts, terminal-based text editors, package creation with automake

  • Date: 2011 / 05 / 16
  • Supervisor: Lothar Richter, Laszlo Kajan
  • Topics: vim, emacs, shell scripts, makefiles, automake, autoconf
  • Toy problem:
    • You have implemented a cool program that reduces sequence redundancy (it consists of multiple files). You told about it to your friend from another lab and now he/she also would like to try it. After trying it he/she likes it so much that she wants the system manager to install it on their computing cluster in a way that everyone in that lab can use it.
    • Problem: in what format to give your friend or the system manager your program? As a system manager, what would you do, how would you handle this request?
  1. Links for preparation:

Programming challenge

Write a short (C / perl / python / shell, as you wish) program that reads text from a file, removes all spaces and writes the result back into a file. Create a distributable tar ball of your program using automake and autoconf. Add a man page as well. Make sure the 'make distcheck' command succeeds on your package. Get familiar with terminal-based text editors. We recommend you implement this programming challenge using vim.

Hints and tips

  1. Edit your package sources list (/etc/apt/sources.list) and enable the 'contrib' and 'non-free' sections of the repository: add contrib and non-free after 'main' on each deb and deb-src line. Refresh the package cache.
  2. Install the 'make', 'make-doc', 'automake' and 'autoconf-doc' packages: these provide automake, autoconf and the info documentation
  3. Learn to navigate the info browser (do info automake, press '?' and read)
  4. Read section 1 Introduction and 2 Autotools Introduction up to and including 2.2.4 Standard Configuration Variables.
  5. Follow the examples (e.g. 'zardoz') in the automake info to create your Makefile.am and configure.ac. You will want to have at least these macros in your configure.ac:
    AC_INIT
    AM_INIT_AUTOMAKE
    AC_CONFIG_FILES
    AC_OUTPUT
    Use the documentation to find out more about these.
  6. You can use the --prefix ./configure option to test the install target at a custom location (e.g. --prefix=/tmp/test)
  7. I recommend you use the pod syntax to create the man page. Install the 'perl-doc' package to gain access to the 'perlpod' manpage. Read: man perlpod; man pod2man.
  8. Create rules in Makefile.am to have make generate the manpage for your script from a .pod source
  9. If your program is a script, use the SCRIPTS primary instead of PROGRAMS
  10. Use the DATA primary to distribute the .pod source and the MANS primary to install the man page
  11. Make sure the .pod source is not installed but the generated man page is (use the automake 'dist' and 'noinst' prefixes as appropriate).
  12. Make sure your package passes the make distcheck test

Advanced challenge

Create a make file (and whatever else program you need) that conducts a blast all-against-all on a set of sequences given in a multi-sequence FASTA file. Think of ways to take advantage say 12 cores in your hardware with the '-j' make option as blast performance usually plateaus at '-a 4' (4 blast threads; asking for more blast threads does not make blast faster). Package your solution with automake/autoconf.

Report

By Maximilian Hecht:

Virtualization with kvm/qemu

  1. Date: 2011 / 05 / 23
  2. Supervisor: Laszlo Kajan
  3. Topics: creating a virtual machine image, starting a virtual machine, using a virtual distributed ethernet VDE) network, using VNC, secure port forwarding with ssh
  4. Toy problem:
    • You want to upgrade the bios firmware of your server. It has neither CD reader or floppy drive. Even though the hardware manufacturer distributes a Linux for doing this, you find out that it does not work. The manufacturer also distributes a DOS firmware upgrade tool. You find a bootable DOS image that you could use to create a bootable DOS USB stick to use to upgrade the firmware. But you need to boot into DOS to format the USB stick.
    • How would you boot your DOS image (be it a floppy image or a cd-rom image)?
  1. Slides:
    1. Comparison of software raid levels on 4 disks File:RAID comparison.pdf
  1. Links for preparation:

Linux Command Exam

There will be a Linux Command Exam today, at the beginning of the practical session. The exam takes 10 minutes and you will get 25 to 30 questions about Linux command lines. You are allowed to use the computer and Internet.

Example test questions

What command would you use to:

  • remove an empty directory
  • remove a potentially filled directory
  • remove all files with '.pl~' extension in a directory tree
  • switch the group write permission on on all files that match the '*.pl' extension in a directory tree
  • list a directory with files sorted on modification time in reverse (newest on bottom)
  • copy a directory tree to another location in an 'archiving' way
  • copy a directory tree to another computer in an efficient way, supposing some of the files are already present on the remote system
  • create the directory /tmp/test/src/linux with one command when only /tmp exists
  • open a man page file in your present directory that is not within the regular man path
  • print your PATH environment variable; what is the function of the PATH environment variable?
  • add your present working directory to your shell search path
  • look at the contents of a text file (name at least two tools)
  • compare two text files
  • list your environment
  • list variables in your environment that are exported
  • kill a process
  • list all 'bash' processes running on your system in user-oriented format
  • temporarily suspend a process
  • resume a temporarily suspended process
  • look at the top processes with respect to memory usage or CPU usage
  • list all ext4 type mounted file systems
  • temporarily mount a fat file system from device sdb1 to a temporary mount point
  • bind-mount /srv/raidarray/project to /srv/nfs4/project
  • eject a cd-rom
  • power off your computer
  • reboot your computer
  • examine the exit status of the last foreground command you executed

Programming challenge

Notes

You all are going to run virtual machines unprivileged. In order to provide proper networking we are going to use VDE: Virtual Distributed Ethernet. Unfortunately as of this time there is no libvirt support for VDE, so you can not use libvirt for managing your virtual machine. You will have to use the hypervisor (kvm) directly.

Since you all have a 4G quota on the virtualization host (i12r-studfilesrv.informatik.tu-muenchen.de), you can not copy over and convert the entire image of your usb sticks.

Challenge

Create a Debian stable 64-bit virtual machine on the virtualization host i12r-studfilesrv.informatik.tu-muenchen.de, configure it according to the table above (IP address, host name, your user name and numeric user ID). By the next session it has to answer pings. You should be able to ssh into it from the host i12r-studfilesrv.informatik.tu-muenchen.de. You should leave the virtual machine running with no active display, but with the possibility to attach a VNC client. Have two shell scripts: one that starts the virtual machine with the default SDL display and one that starts with VNC but with no actual display attached (-vnc none).

Virtual machine resource limits:

  • 512M memory
  • 1 core
  • 2+ GB disk image
  • VDE socket: /var/run/kvm0.ctl
  • A dhcp (and name) server is provided for the 192.168.16.0/24 network for your virtual machines with NAT to the Internet
  • Gateway: 192.168.16.1, netmask: 255.255.255.0

Please do not exceed the memory limit or we are going to have serious problems! The 2G disk image should be sufficient as long as we install only what is necessary. If you have space to spare, create a bigger image if you want. I can grant bigger quotas if you explain to me that you really need it.

Hints and tips

  1. Begin with creating a new virtual machine image of the desired size, I recommend the qcow2 format (kvm-img)
  2. Download the appropriate (small) CD install image
  3. Boot the CD install image with the KVM hypervisor and install Debian onto the image file (so you will need to specify a drive for that as well) - use the kvm command
    • You will want to be on a graphical screen and have X11 forwarding for your session when you do this.
    • You have to configure the hypervisor for VDE networking using the socket above (-net vde)
    • Give the virtual machine a network device (-net nic) that is connected to the same VLAN the VDE switch is on. I recommend using the 'virtio' model for best performance. Give a MAC address to your virtual network interface that is unique among the course participants.
  4. Either configure the network statically with the IP address and host name given in the table above or change the configuration later in /etc/network/interfaces
  5. Install the ssh server.
  6. After the installation reboot the virtual machine from the new machine image. Write the two shell scripts.
  7. Change your numeric user ID according to the table above and remember to change the ownership of your files
  8. Once you are confident about your virtual machine experiment with the VNC solution
    1. In order to be able to start a VCN session with -vnc none you will need to provide a way to monitor the hypervisor. I recommend using a UNIX socket like this:
      • -chardev socket,id=monitor,path=<your_home_dir>/tbl2011.monitor,server,nowait -mon chardev=monitor,mode=readline
    2. Start the virtual machine with -vnc none and detach it from the terminal (nohup)
    3. Connecting to the monitor socket <your_home_dir>/tbl2011.monitor with 'nc -U' and start a VNC session with the 'change vnc :<digit>' command
    4. Forward the port (5900+<digit>) with ssh and connect to it from your remote machine (vncviewer)

Advanced challenge

  • What would it take to migrate your virtual machine from i12r-studfilesrv to another host? Using libvirt or not?
  • Implement VDE support for libvirt - this would be welcome my many I am sure.

Report

By Veit Höhn

User management / directory services

Programming Challenge

  • Set up a directory service (LDAP) for the practical and define a fitting directory structure. Use 'cd=course' as the base DN of your directory.
  • Check the monitoring tools and log files for your installed directory service.
  • Connect the user management of your debian installation to your LDAP - remove your user and group entries from the files
  • Read out of the above table your numeric user id and change it (also the numeric group id) in your LDAP database (and password and group files if you had not done so yet)
  • Change the ownership of any file owned by the old user id. You will have to update the owner and group to the new number on files most likely located in /home/<username> (if you had not done this yet)
  • Configure your LDAP server as a replication provider
  • Learn to understand LDAP access control statements (olcAccess, man slapd.access(5)). Interpret the olcAccess statements that come with the default configuration.

Advanced challenge

  • Secure the connection to the LDAP server with TLS or SSL and a server certificate (we recommend 'tinyca2' for certificate management)
  • Configure your LDAP server as a replication provider so your fellow course members can pick up your user records:
    • Read these man pages: syncrepl section in slapd.conf, slapo-syncprov
    • Configure the syncprov overlay (I think two extra lines in your slapd.conf)
    • Look out: slapd is now configured with the special 'slapd-config' configuration backend. We recommend to configure it with 'ldapvi -b cn=config'

Laszlo's hints and tips

The goal: have regular users' accounts maintained in the LDAP directory.

  1. apt-get install slapd libnss-ldap nscd
  2. Edit /etc/hosts, have your IP address associated with your host name (see above table), like: 192.168.16.X <NAME>.course <NAME>
  3. Edit /etc/default/slapd, have at least 'ldapi:///' on the SLAPD_SERVICES line. This gives makes your LDAP server listen for connections on a Unix socket.
  4. Use the slapd-config man page for reference on the configuration options
  5. Stop the ldap server, change /etc/ldap/slapd.d/cn=config/olcDatabase={0}config.ldif and set a password for the root account (you can see its DN on the olcRootDN line). Use slappasswd with the SSHA hash. Restart the ldap server.
  6. Use ldapvi -b cn=config -D [the DN you saw on the olcRootDN line, for me cn=admin,cn=config] to edit the configuration of the running ldap server. Changes are permanent.
    Here you can also see the schema definitions of object classes. Use this to figure out what attributes are mandatory when you add your user and group (below).
  7. Use 'ldapvi --discover -D cn=admin,dc=course -h ldapi:///' to
    • add two organizational units ou=people,dc=course and ou=group,dc=course
    • add your group as an 'objectClass: posixGroup': cn=<usr>,ou=group,dc=course
    • add your user as an 'objectClass: posixAccount; objectClass shadowAccount; objectClass inetOrgPerson': uid=<usr>,ou=people,dc=course
    • use slappasswd to generate the encrypted form of your password
    • Remember to set the 'shadowLastChange' attribute or your account is not valid and you can not log in. This is the number of days elapsed since the Epoch, setting it to 1 is sufficient. You can use 'date' to get the number of seconds since the Epoch and then by simple division the number of days, if you want.
  8. It is not usually necessary, but it may be a good idea to invalidate the name service cache kept by the 'nscd' daemon after modifying user and group attributes: nscd -i passwd; nscd -i group. During testing you can altogether stop the nscd daemon.
  9. dpkg-reconfigure libnss-ldap libpam-ldap
  10. Edit /etc/nsswitch.conf, append 'ldap' to the passwd, group and shadow databases' lines
  11. Use 'pam-auth-update' to enable LDAP for PAM (authentication, etc.)
  12. Have a look at /etc/pam.d/common-*, see how it works
  13. Use 'getent passwd' and 'getent group' to verify that you LDAP connection to name services works. If you still have your user and group defined in /etc/passwd and /etc/group (and their shadow) files, you should see your user and group entry listed twice by the getent commands. The second is the one that comes from your LDAP server since 'ldap' appears second (to 'files' or 'compat') in /etc/nsswitch.conf.
  14. If you see that getent returns the right records for yourself and your group, remove your user and group entry from /etc/{passwd,shadow,group,gshadow}. Try to log into your virtual machine as yourself to test that it works.
  15. Configure your ldap server as a replication provider with the syncrepl overlay: man slapo-syncprov
    1. Make slapd load the 'syncprov' module. Stop the ldap server (it does not see to allow dynamic configuration of the list of modules), edit cn=config/cn=module{0}.ldif, add another olcModuleLoad attribute for syncprov and restart the server.
    2. Add a new olcSyncProvConfig entry for the 'syncprov' overlay, making it a child of the database entry 'olcDatabase={1}hdb,cn=config'. Set the olcSpReloadHint attribute to TRUE as suggested on the man page slapo-syncprov.
  16. After the next session we are going to configure our LDAP servers to act as replication clients for each other's servers so that we all can log in to each other's virtual machines.
    1. Add the olcServerID attribute to cn=config, set it to your offset in the host names table
    2. Add an LDAP user 'replclient' (say in dc=course) that is allowed to /read/ your user and group entries in order to allow other course members to read your LDAP database using this user
    3. Add access rights to the above user to your user and group entries (add 'olcAccess' attributes to 'olcDatabase={1}hdb,cn=config' - man slapd.access. An example: 'olcAccess: {1}to dn.sub="ou=people,dc=course" by dn="cn=replclient,dc=course" read by * break')
    4. Configure replication of another course member by adding the olcSyncrepl attribute to olcDatabase={1}hdb,cn=config, e.g. to replicate Laszlo's user and group entries: 'olcSyncrepl: {0}rid=2 provider=ldap://192.168.16.2 searchbase=dc=course type=refreshAndPersist retry="60 +" filter=(|(uid=lkajan)(&(cn=lkajan)(objectClass=posixGroup))) bindmethod=simple binddn=cn=replclient,dc=course credentials=guest'
    5. Add olcMirrorMode: TRUE to olcDatabase={1}hdb,cn=config in order to allow local updates
    6. You should restart your LDAP server after adding the new replication configuration or weird errors will occur
    7. Entries that were created/modified before the replication is configured do not always seem to be copied over to the replication client. However if they are modified on the replication provider they do get replicated.
  17. In order to debug the LDAP server:
    1. Stop the nscd daemon
    2. Start the LDAP server from the command line with '/usr/sbin/slapd -h 'ldap:/// ldapi:///' -g openldap -u openldap -F /etc/ldap/slapd.d -d 16640'

Markus' hints and tips from last year

  • Only make changes in /etc/ldap
  • You will need these packages: slapd ldap-utils migrationtools
  • Save all the original configurations, e.g. in XXX.old
  • Use ldapsearch to test your ldap server from the command line
  • You will need these packages: ldap-utils libpam-ldap libnss-ldap nscd

Report

by Michael

Mail, DNS

Vi Exam

There will be a Vi exam today at the beginning of the practical session. The exams takes 10 minutes and you will get 25 to 30 questions about vim. You are allowed to use the computer and Internet.

Example test questions

  • How do you start vi?
  • How do you start vi and automatically open a file at line 22?
  • What is the command to save a file?
  • How do you quit vi without saving the file?
  • What is the command to jump to line 33?
  • What is the command to jump to the middle row of your current window?
  • What is the command to delete a complete line?
  • What is the command to delete one word (including the spaces)?
  • How do you copy a word?
  • What is the command to replace all name '/tmp' to '/var/tmp'?
  • How do you open a new file?
  • What is the command to delete the next 10 lines?
  • What is the command to move the courser to the next occurrence of 'tmp'?
  • What is the shortcut for undo?
  • Which key stoke gives you information (e.g. number of lines) about your file?

Packages recommended for installation

  • Please install these packages without recommendations (check out 'Install recommended packages automatically' in the preferences in aptitude):
    • gnome-core
    • xorg
    • iceweasel, icedove

Programming challenge

  • Configure your LDAP server to be reachable for other course members via the network (without SSL/TLS)
  • Create a new LDAP user that is going to be used for replication by replication clients. Allow this user to read user and group records.
  • Configure your LDAP server to replicate the other course members' user and group records (but no more - mind the search filter)
  • Set up and configure a DNS server (bind9 recommended)
  • Set up and configure a mail server (postfix recommended)
  • Set up an IMAP server (dovecot recommended)
  • Use Thunderbird / Icedove to send a mail to another course member, configure the address book in Icedove/Thunderbird to connect to your LDAP server

Hints and tips

  • packages to install: bind9, dnsutils; postfix, postfix-doc, bsd-mailx; dovecot-imapd; icedove; ca-certificates; procmail; tinyca2 (advanced challenge);
  • Postfix configuration: choose 'Internet site'

Name server

  • Edit /etc/bind/named.conf.local, add:
zone "course" {
       type master;
       file "/etc/bind/db.course";
};

zone "16.168.192.in-addr.arpa" {
       type master;
       file "/etc/bind/db.192.168.16";
};
  • Edit /etc/bind/db.course and /etc/bind/db.192.168.16, have:

/etc/bind/db.course:

;
; BIND reverse data file for broadcast zone
;
$TTL    86400
@       IN      SOA     lkajan.course. root.lkajan.course. (
                       10051701         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      lkajan.course.

lkajan          A       192.168.16.2
<other course members>

/etc/bind/db.192.168.16:

;
; BIND reverse data file for broadcast zone
;
$TTL    86400
@       IN      SOA     lkajan.course. root.lkajan.course. (
                       10051701         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      lkajan.course.

2       PTR     lkajan.course.
<other course members>
  • Replace lkajan and 192.168.16.2 with your host name and IP as appropriate
  • Test-load your named configuration: named-checkconf -z
  • Restart the name server
  • Update your /etc/resolv.conf with your own name server:
search course
nameserver 127.0.0.1
...
  • Test the name server with: host <name>.course; dig <name>.course; ping <name>.course

Mail server

  • Edit /etc/postfix/main.cf, review the mydestination line as well as mynetworks. Add your IP address to the mynetworks list.
  • Restart postfix and check that all is well with it (check the logs)
  • Add a root alias to your regular account (/etc/aliases); recreate the alias database
  • Edit your ~/.procmailrc and configure Maildir / mbox delivery as you prefer
# Maildir:
DEFAULT="$HOME/Maildir/"
  • Send a mail to yourself as root
  • Examine the mail log and check if the mail was delivered well

Dovecot (IMAP server)

  • Edit /etc/dovecot/dovecot.conf
  • Do not change anything but look at the protocols and the authentication: PAM does the work for us; ssl_cert_file and ssl_key_file: this is where we are going to secure communication to the server

Thunderbird / Icedove

  • Start Icedove and configure a new mail server:
    • Email address: <username>@<hostname>.course
    • Type: IMAP
    • Incoming server: <hostname>.course or 127.0.0.1
    • Outgoing server: <hostname>.course or 127.0.0.1
    • Configure LDAP: Preferences -> Composition -> Addressing -> Directory server -> Edit directories -> Add:
Hostname: localhost
Base DN: dc=course
Port n: 389
Bind DN: uid=<username>,,ou=people,dc=course
    • Make sure your LDAP server serves connections to ldap://localhost/ (check in /etc/default/slapd
    • Try sending a mail to another course member, e.g. Laszlo Kajan <lkajan@lkajan.course>

Advanced challenge

  • Set up procmail recipes that automatically:
    • reply to the sender that you are busy preparing to an exam if the mail subject contains the word 'work'
    • reply to the sender that you are busy with your work when the subject contains the work 'exam'
    • reply to the sender that you are ill when the subject contains both 'exam' and 'work'
  • Create a postfix regular expression table for aliases and use this table to deliver all mail matching the pattern '/^sink/' to /dev/null
  • Configure spamassassin for your MTA (postfix) or in .procmailrc

Report

By Antonia Stank:

Web server

  1. date: 2011 / 06 / 20
  2. supervisor: Stefan Seemayer
  3. topics: Apache, CGI, PHP
  4. links for preparation:
  5. Preparation questions:
    • Assume you wanted to get an officially authenticated SSL certificate for one domain and one year. How much would that cost you / your organization?
    • What is the advantage of self-signed certificates compared to using no certificates and no encryption? What is the disadvantage of self-signed certificates in comparison to certificates signed by an official certificate authority (CA)?
    • What is the idea of hardening? How could you reduce the attack surface of your apache installation?

Programming Challenge

  • Install Apache and learn about virtual hosts
  • Create a simple website and make it reachable by other course members
  • Set up a HTTPS connection to a website
  • Set up PHP for use with Apache and create a phpinfo page.
  • Experiment with different methods of access control
    • Using IP adresses
    • Using username/password authentication
  • Install phpldapadmin and get it working with your LDAP server
  • Install ldap-account-manager and get it working with your LDAP server

Apache

  1. Install package: apache2
  2. Check if apache is installed correctly by pointing your browser to http://localhost (or the IP address of your server)
  3. Modify the default website to make it a little more personal. The current root of your apache webserver is located in /var/www/
  4. Familiarize yourself with the apache process manager located at /etc/init.d/apache2
    • How would you reload a modified configuration file?
    • How do you find out at which PID apache is running?
    • What is a graceful stop, compared to a regular stop?
  5. Familiarize yourself with the default configuration at /etc/apache2/ .
    • What user (and group) is apache running as?
    • Where would you find error messages?
    • Where would you put user-defined global configuration?
    • What modules are available? Which ones are enabled?
    • You can use aptitude search apache2-mod- to list installable modules.
    • You can use a2[en|dis]mod to enable/disable modules and a2[en|dis]site to enable/disable sites.
  6. Create a new site at sites-available/ to make your website accessible by its hostname. E.g.:
    NameVirtualHost *
    <VirtualHost *>
     
     ServerName seemayer
     ServerAlias seemayer.course seemayer.course
    
     DocumentRoot /var/www
     
     ServerAdmin webmaster@seemayer.course
     
     # Logfiles:
     CustomLog /var/log/apache2/access_seemayer.log combined
     ErrorLog /var/log/apache2/error_seemayer.log
     LogLevel warn
     
     <Location />
       Options Indexes FollowSymLinks MultiViews
       Allow from all
     </Location>
    
    </VirtualHost>
    

  7. Use a2ensite to enable your newly created site and re-load the apache configuration.
  8. To make sure apache is using the right site configuration, disable the default and default-ssl sites using a2dissite.
  9. Your site should now be available from its hostname, e.g. http://seemayer/.

HTTPS

  1. Create and sign a SSL certificate for the server:
    mkdir /etc/apache2/ssl
    cd /etc/apache2/ssl
     
    openssl req -new -x509 -nodes -out seemayer.course.crt -keyout seemayer.course.key
    
    Country Name (2 letter code) [AU]:DE
    State or Province Name (full name) [Some-State]:Bavaria
    Locality Name (eg, city) []:Garching
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:Rostlab
    Organizational Unit Name (eg, section) []:
    Common Name (eg, YOUR name) []:Stefans Webserver
    Email Address []:seemayer@rostlab.org
    

    Alternatively, you can use the GUI interface tinyca for certificate management.

  2. Change the virtual host:
     
    # You may or may not need this line - experiment!
    Listen [youripadress]:443
    
    <VirtualHost [youripadress]:443>
     ServerName seemayer.course
     DocumentRoot /var/www
      
     SSLEngine On
     SSLCipherSuite HIGH:MEDIUM
     SSLCertificateFile	/etc/apache2/ssl/seemayer.course.crt
     SSLCertificateKeyFile	/etc/apache2/ssl/seemayer.course.key
      
     # ... other settings omitted here ...
     
    </VirtualHost>
    
  3. Enable the Apache SSL module using a2enmod, then restart apache to load the SSL engine.
  4. Go to https://yourhostname/ to check out the now SSL-secured website.
  5. You can add a redirection to the SSL Version of your website when the user specifies its HTTP adress. Just add the following to the bottom of your site file:
    <VirtualHost [youripadress]:80>
     ServerName seemayer.course
     <Location />
      Redirect permanent / https://seemayer.course/
     </Location>
    </VirtualHost>
    

PHP

  1. Install the PHP5 module for apache (see above to find out how to determine the debian package to install).
    • You will be asked to remove the package apache2-mpm-worker. This is because PHP is not compatible with the faster worker-MPM, so it has to be replaced with the slower prefork-MPM. The package manager should handle things automatically, though.
    • The package installation will automatically enable the apache module. Restart apache to load it.
  2. Create a PHP test page. The easiest and most verbose testpage is available from creating a file /var/www/test.php with the contents:
    <?php phpinfo(); ?>
    
  3. Point your browser to the new test page. What information is being displayed from the PHP information page?

Access Control

  1. Access control by IP Address
    • In your site configuration, where it says Allow from all, replace with one of the following:
       # Single IP-adress: 
       Allow from 192.168.16.5 192.168.16.15 
      
       # IP range
       Allow from 192.168.16.
      
    • Read up on Subnet masks and the mod_access documentation for more examples on filtering by IP address.
  2. Access control by Username/Password
    1. Create a htpasswd file with an entry for the user MyUserName:
      htpasswd -c /etc/apache2/htpasswd MyUserName
      
      • How would you add another user?
      • How would you remove a user from the htpasswd file?
    2. Instead of the Allow from and ' in your site file, put the following:
      AuthType Basic
      AuthName "Internal area - Authorized users only"
      AuthUserFile /etc/apache2/htpasswd
      Require valid-user
      

Advanced challenges

  • Experiment with advanced access control methods
    • Let users in your LDAP server log in
    • Let users log in with certificates
  • Enable user homepages using mod_userdir
  • Enable compression of output using mod_deflate

Materials and methods

  • Session1_debriefing.pdf

Report

By Markus Meier:

Databases and SQL

  • date: 2011 / 06 / 27
  • supervisor: Tanya Goldberg
  • topics: DBMS, MySQL, postgreSql, SQL
  • links for preparation:
  • Slides:

File:InroSlides DatabasesMySQL.pdf

Programming Challenge

  • Install and configure a MySQL Database server.
    • There should be a database with your name and a user with your name_db.
    • Only the user should have access to the database from localhost.
  • Create a table in your database and fill it with some data.
    • For example use PHP or Perl.
  • Create a backup from your database.
  • Install phpMyAdmin to provide a nice web-based interface for users.

Hints and tips

mysql

1. Install package: mysql-server

  • Which additional packages will be installed?
  • Which client programs?

2. Set a MySQL root password during the installation process

3. Familiarize yourself with the default configuration file /etc/my.cnf

  • What is the TCP/IP Port the MySQL server/ client applications will listen to?
  • How many concurrent sessions the MySQL server will allow?
  • What is the size of the query cache used to cache SELECT results?
  • Replace the content of /etc/my.cnf with the one from my-small.cnf

A nice GUI based MySQL client is the 'mysql-navigator'

MySQL Administration

At this moment, we will create basic permissions for a user and database: allow access to localhost to all databases, and a computer which is also on the network, e.g "goldberg.course" will have access to all databases.

If you haven't done this already, set the root password for MySQL

mysqladmin -u root password your_new_password

Connect to your MySQL server

mysql -u root -p

To access the user, host databases, etc. type this:

mysql> use mysql 

To give localhost permission to access all databases, enter this:

mysql> insert into 
        -> host(host,db,Select_priv, Insert_priv, Update_priv, 
        -> Delete_priv, Create_priv, Drop_priv)
        -> values('localhost','%','Y','Y','Y','Y','Y','Y');
  • the '%' can be replaced with a database name. The '%' is a wildcard.

To allow access from another hostname (in this case "goldberg.course") add this:

mysql> insert into 
 -> host(host,db,Select_priv, Insert_priv, Update_priv, 
 -> Delete_priv, Create_priv, Drop_priv)
 -> values('goldberg.course','%','Y','Y','Y','Y','Y','Y');

To create a user 'tanya' who can access the MySQL server from localhost, type this:

mysql> insert into 
        -> user (host, user, password)
        -> values('localhost','tanya',password('XXX'));

To give the user access from another hostname, domain, etc. add other entries accordingly. For example, to give user 'tanya' access from goldberg.course :

mysql> insert into 
        -> user (host, user, password)
        -> values('goldberg.course','tanya',password('XXX'));

To give the user permissions to access a database from localhost, add this entry and change with your appropriate information:

mysql> insert into
     -> db (host,db,user,Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv)
     -> values ('localhost','tanya_db','tanya','Y','Y','Y','Y','Y','Y');

To exit the mysql server print

mysql> quit;

Finally, create the actual database (in this case, 'tanya_db') type this:

mysqladmin -u root -p create mydatabase
  • After prompting you for a password, it should create the database.

At this point, you must reload MySQL. Type:

mysqladmin -u root -p reload

Much more you can find at http://dev.mysql.com/doc/refman/5.0/en/index.html

Now, create a database with your name and a user with your name. Only the user should have access to the database from localhost.

Writing and Reading

We are going to use Perl to write into the database.

  • Install 'perl' and a module to connect to a database: libdbi-perl
  • Create a table for your SQL queries. Populate it with some data and retrieve it afterwards.
    • Some pseudo perl code
use DBI;
my @con = ('DBI:mysql:cgdiso:tanya_db', 'tanya', 'XXX'); 
my $dbh = DBI->connect( @con ) || die "Database connection not made: $DBI::errstr";
# create table
$sql = "CREATE TABLE disorder (
 Id VARCHAR(12) NOT NULL,
 Residue LONGTEXT NOT NULL,
 MD LONGTEXT NOT NULL,
 time DATETIME NOT NULL,
 PRIMARY KEY id (Id));
$sth = $dbh->prepare( $sql );
sth->execute();
$sth->finish();
# CLOSE connection
$dbh->disconnect();
  • Can you do the same with any other programming language?

Backup

There are several general methods for making backups.

  • Use 'mysqldump' to create a backup from your databases.
  • What are the other methods for making backups?
  • What is the statement to check a table for errors?
  • When an error is found, what is the command to fix it?

phpMyAdmin

Install the package: phpmyadmin

  • you have to choose a webserver during the installation process: apache2
  • you have to add this new website to your apache configuration /etc/apache2/apache2.conf
Include /etc/phpmyadmin/apache.conf
    • What is included in /etc/phpmyadmin/apache.conf?
    • Do not forget to reload the apache
  • go to http://localhost/phpmyadmin and login
  • now we can do everything (?) with the webinterface
    • can you do all the changes we did with the command line?

Advanced challenge

  • Install and configure PostgreSQL
  • Install a nice user front-end
  • Use your preferred programming language to access the database

Report

By Evi Berchtold:

Basic web applications - CMS and Wiki

  • date: 2011 / 07 / 04
  • supervisor: Tanya Goldberg
  • topics include: CMS, Wiki, Bug tracking, Forum, Calendar
  • links for preparation:
  • Slides

File:IntroSlides WikiCMS.pdf

SQL Exam

There will be an exam today at the beginning of the practical session. The exam will take 30-35 minutes and you will get 20 questions about SQL and MySQL. You are allowed to use computer and the Internet.

Example test questions

1. What is SQL? What is MySQL?

2. How do you start MySQL on Linux?

3. What is the default port for MySQL server?

4. State two security recommendations while using MySQL.

5. How do you use mysqldump to backup your database?

6. A table has a column defined as a TIMESTAMP. What happens if the row gets altered?

7. Which privileges tables would you modify to grant user 'user' access to the database 'students' from 'localhost'?

8. How do you check/ upgrade tables in InnoDB?

9. How do you change password for an existing user (e.g. root) via mysqladmin?

10. A column is set to AUTO INCREMENT. What happens if you reach the maximum value for that table?

11. The MySQL server is located by default in a directory called

a. docs
b. lib
c. scripts
d. bin

12. The MySQL option --i-am-dummy stands for...

a. what it sais
b. preventing from unwanted modifications on the database
c. preventing from doing other things then logging in and out
d. none of the above

13. Which of the following CREATE TABLE statements is correct?

a. Create table student
studentID int not null auto-increment primary key, name varchar(20)
type=InnoDB;
b. Create table student type=InnoDB
(studentID int not null auto-increment primary key, name varchar(20)
);
c. Create student
(studentID int not null auto-increment primary key, name varchar(20)
) type=InnoDB;
d. Create table student
(studentID int not null auto-increment primary key, name varchar(20)
) type=InnoDB;

14. How do you select line numbers 16, 17, 18, 19 and 20 via a LIMIT clause?

a. LIMIT 15,20
b. LIMIT 14,5
c. LIMIT 15,5
d. LIMIT 15,6

15. CHECK TABLE reports: "Table is already up to date". Meaning for you that

a. you have to start REPAIR TABLE
b. you have to restart CHECK TABLE
c. your table is OK
d. none of the above

Programming Challenge

  • Install a wiki engine (e.g. MediaWiki).
  • Install the CMS of your choice (e.g. Typo3).
  • Connect the user management of the CMS with your LDAP.
  • Create a simple web page with your CMS for the practical.

Please send Laszlo and Tanya (goldberg@rostlab.org) screenshots, including the URL of the browser, of

  • one page in your wiki.
  • the front- and backend from your CMS and
  • 2 pictures from different subpages in your CMS.

Hints and tips

wiki

There are many different wiki engines:

We are going to use MediaWiki (http://www.mediawiki.org), one of the most popular wiki engines available.

1. Install a good and stable debian package: mediawiki

2. Adjust the MediaWiki configuration file to the system environment

  • add to your virtual host file
Include /etc/mediawiki/apache.conf 
  • do not forget to reload the Apache
  • which domain do you now use to access the mediawiki?
  • Uncomment the third line in /etc/mediawiki/apache.conf, so that line reads
Alias /mediawiki /var/lib/mediawiki
  • the alias can be replaced with any other alias you want

3. Complete the installation settings over the Internet

  • discuss with your neighbor a suitable configuration

4. Review the settings in the default and the local configuration files

  • the default configuration file should not be edited
  • what permissions do you set for the 'LocalSettings.php' file?'

5. Modify the main page to make it a little more personal and at least add a logo. Allow registered users to change the content ($wgGroupPermissions).

CMS

There are many different CMS implementations.

We are going to use Typo3 (http://www.typo3.org).

1. Install a good and stable debian package: typo3

2. Add to your virtual host file

Include /etc/typo3-dummy/apache.conf 
  • Which domain do you now use to access the typo?

3. Now you can access your Typo3 installation, for example from http://localhost/cms/typo3/install

  • To get into the installation process you have to create an empty file (ENABLE_INSTALL_TOOL) in the /var/lib/typo3-dummy/typo3conf directory.
  • Follow the installation instructions
    • Discuss a suitable configuration with your neighbor.
    • You have to create a new db!
    • Do not remove the typo3/install folder and the ENABLE_INSTALL_TOOL file. Do it after the complete configuration!

4. Login into typo3 with username 'admin' and password 'password' and make the finale configuration

5. Create a web page

  • Create a simple page object with some simple content objects.
    • Do not forget to select a template. Otherwise your web page won't be shown
  • Try to build up a page tree structure. If you choose a good layout, you can recover the page tree structure in the navigation.
  • Which URL do you use to access the front-end and which for the back-end?
  • To connect the back-end to the LDAP you need an extension 'eu_ldap'. For the configuration see the manual (at the typo3.org page)
  • Now you need a Layout Template. You can use 'TemplaVoila' and build your own one. Or google the web for some example templates to include them.
  • After several configuration you have to clean the CACHE to see the changes! (top right)

5. Add some nice extensions (you can find them at typo3.org):

  • realurl: for nicer URLs
  • tt_news: to present news in a nice layout
  • ...
    • Now you can remove the typo3/install folder and the ENABLE_INSTALL_TOOL file

Advanced challenge

Wiki Advanced Challenge

CMS Advanced Challenge

  • Integrate an internal user section (connection to the LDAP and use https) for the front-end! There is an (several) extension available for Typo3.

Report

By Khadija El-Amrani:

By Erik Pfeiffenberger:

Modules in programming, development and design

  1. Date: 2011 / 07 / 11
  2. Supervisor: Laszlo Kajan
  3. Topics: monolithic programming vs. modular programming in practice, software design exercise, successful packaging practices, packaging for Debian/Ubuntu, packaging for RPM-based distributions
  4. Links for preparation:

Programming Challenge

  • analyze provided example case and implement solution
  • package the resulting software

Hints and tips

Report

N.a.

Computer clusters and external services : trip to the LRZ

  1. date: 2011 / 07 / 25
  2. topics: computer cluster hardware, batch systems
  3. links for preparation:

Programming Challenge

We will get a guided tour in the LRZ to see all the server and services in live operation.

Hints and Tips

  • We meet at 2 p.m. in the rostlab!
  • Please be in time.
  • A photo identification is required.
  • There is air condition in the server rooms. Please dress appropriate.


Cloud services (incl. Amazon Web Services)

  1. date: postponed due to technical difficulties
  2. topics: Amazon EC2, Amazon S3, Amazon SimpleDB, Amazon RDS, Amazon SQS, Amazon CloudFront and Amazon Elastic MapReduce.
  3. links for preparation:
  1. You will get an AWS voucher about 100USD. Please be careful, everything you will use more than that you have to pay by yourself!

Programming Challenge

XXX Amazon Elastic Block Store Public Data Sets

Hints and tips

XXX

Advanced challenge

XXX

Materials and methods

  • Session1_debriefing.pdf

Report

By XXX XXX:

  • slides.pdf
  • protocoll.pdf
Personal tools