The bioinformatics lab 2010

From Rost Lab Open

This practical is a hands-on training that will make you successful in a Bioinformatic lab! All the services and details from bottom up, e.g. Wiki, version control, compute cluster, software packaging

More details can be found at the http://rostlab.informatik.tu-muenchen.de/cms/biolab2010/ website.

Contents

External media / installing a Linux OS / Debian stable / Linux kernel

  1. date: 2010 / 04 / 19
  2. responsible: Laszlo
  3. topics: Linux distributions, rpm vs. deb, stable vs. bleeding edge; decisions to make at the time of system installation. Partitioning, LVM, cryptsetup; kernel modules, boot loader, initramdisk; dpkg, apt, aptitude, apticron; sysctl, /proc, /sys
  4. links for preparation:

Programming challenge

Install and configure a Debian stable base system on your USB stick. Be careful with the partition, only use the USB device. Do not forget to enable boot on USB in your BIOS. Get familiar with vim.

Hints and tips

  1. Choose English so Laszlo can help
  2. Choose manual partitioning and make sure you leave alone all drives except the USB stick
  3. Note the device name of the USB stick, something like 'sda', 'sdb'...
  4. Have 3 partitions on the USB stick: one for a /boot file system which should be bootable and not more that 512 MB; one for swap space - 512MB; one for a / (root) file system. ext3 is a good choice for the /boot and / file systems.
  5. There is no proxy for accessing the internet
  6. Choose NOT to install the boot loader into the first hard drive: 'Install the GRUB boot loader to the master boot record?' NO
  7. Install the boot loader to the device you noted at step 3 above, e.g. /dev/sdb. Be careful not to install GRUB onto your hard drive.
  8. Boot from the USB key. Chances are this is going go fail because the way the installer installs GRUB (the boot loader) is not the best for removable devices.
  9. If booting fails with Error 17: Can not mount selected partition - the error we observed so far - hit 'e' on the grub screen to edit the first menu option. Hit 'e' again to edit the line root (hdX,Y). 'X' represents the hard drives as the BIOS sees them, numbered from 0. Error 17 indicates the the presently set hard drive does not contain a /boot partition where indicated by the 'root' line, so experiment with other numbers, e.g. 0: root (hd0,0). Do not change the second number 'Y': that indicates the partition and is likely to be correct. Try booting with the new 'root' line (hit 'b'). This modification is not permanent.
  10. Getting the 'root' line right should enable you to boot into Linux. The permanent solution is to use GRUB2 instead of GRUB. GRUB2 uses the universally unique identifier (UUID) to indicate root and boot partitions and so can handle removable devices well.
  11. Once you booted Linux log in as root and replace GRUB with GRUB2. You are going to need a working network connection for this step (courtesy of Fabian). Start the package manager 'aptitude'. Hit '?' and learn the keys' meanings. Search for the package 'grub2' and mark it for installation. Install it. Answer NO to the question whether to chain-load GRUB2. When the installation is complete exit aptitude.
  12. Install this version of grub to the master boot record of the USB key: execute update-grub. Find the current device name of the usb key, e.g. with mount, looking for a line like /dev/sda3 on /boot type ext3. In this case the USB stick is /dev/sda. Execute grub-install with the device for the USB key, e.g. grub-install /dev/sda.
  13. Reboot from the USB key - this time it should boot cleanly into Linux.

Advanced challenge

Install Debian into an encrypted partition.

Materials and methods

Report

By Stefan Seemayer:

Shell scripts, command line (terminal) editors and compilation

  1. date: 2010 / 04 / 26
  2. responsible: Markus
  3. topics: vim, emacs, shell scripts, gcc, makefiles, automake, autoconf, kernel and module compilation
  4. links for preparation:

Programming challenge

Write a short C / perl / python program that reads text from a file, removes all spaces and writes the result back into a file. Create a distributable tar ball of your program using automake and autoconf. Add a man page as well. Get familiar with terminal-based text editors. We recommend you implement this programming challenge using vim.

Advanced challenge

Compile the kernel of your choice making your hard drive, video card and network card drivers compiled in (not in modules). Do the kernel compilation on an in-memory file system (such as tmpfs) - doing it on the USB stick would take forever.

Hints and tips

Automake and autoconf

  1. Edit your package sources list (/etc/apt/sources.list) and enable the 'contrib' and 'non-free' sections of the repository: add contrib and non-free after 'main' on each deb and deb-src line. Refresh the package cache.
  2. Install the 'make', 'make-doc', 'automake' and 'autoconf-doc' packages: these provide automake, autoconf and the info documentation
  3. Learn to navigate the info browser (do info automake, press '?' and read)
  4. Read section 1 Introduction and 2 Autotools Introduction up to and including 2.2.4 Standard Configuration Variables.
  5. Follow the examples (e.g. 'zardoz') in the automake info to create your Makefile.am and configure.ac. You will want to have at least these macros in your configure.ac:
    AC_INIT
    AM_INIT_AUTOMAKE
    AC_CONFIG_FILES
    AC_OUTPUT
    Use the documentation to find out more about these.
  6. You can use the --prefix ./configure option to test the install target at a custom location (e.g. --prefix=/tmp/test)
  7. I recommend you use the pod syntax to create the man page. Install the 'perl-doc' package to gain access to the 'perlpod' manpage. Read: man perlpod; man pod2man.
  8. Create rules in Makefile.am to have make generate the manpage for your script from a .pod source
  9. If your program is a script, use the SCRIPTS primary instead of PROGRAMS
  10. Use the DATA primary to account for the .pod source and the MANS primary for the man page
  11. Make sure the .pod source is not installed but the generated man page is (use the automake 'dist' and 'noinst' prefixes as appropriate).
  12. Make sure your package passes the make distcheck test

Kernel

  1. Mount (permanently if you want) a tmpfs on /tmp: 'man mount; man fstab'. Compiling a kernel with IO to the USB stick would take forever. You should have enough memory to hold the entire kernel compilation in memory. A 2GB tmpfs will be enough if you un-check the Kernel hacking -> Kernel debugging option (1166 MB was used in my case)
  2. I advise you use make menuconfig to configure the kernel of your choice
  3. You can use the lspci -v command to find out about the driver of your network interface. Use this page http://www.debian.org/distrib/packages to find out what package provides lspci
  4. Remember to create an initramdisk for your compiled kernel in case you wish to test it. man update-initramfs

Materials and methods

Report

By Florian Seifert:

Linux Command Exam: Monday, 10 / May / 2010

There will be a Linux Command Exam at Monday, 10th of May, at the beginning of the practical session. The exams takes 10 minutes and you will get 25 to 30 questions about Linux command lines. You are allowed to use the computer and Internet.

Example test questions

What command would you use to:

  • remove an empty directory
  • remove a potentially filled directory
  • remove all files with '.pl~' extension in a directory tree
  • switch the group write permission on on all files that match the '*.pl' extension in a directory tree
  • list a directory with files sorted on modification time in reverse (newest on bottom)
  • copy a directory tree to another location in an 'archiving' way
  • copy a directory tree to another computer in an efficient way
  • create the directory /tmp/test/src/linux with one command when only /tmp exists
  • open a man page file in your present directory that is not within the regular man path
  • print your PATH? What is the function of the PATH environment variable?
  • add your present working directory to your path
  • look at the contents of a text file (name at least two tools)
  • compare two text files
  • list your environment
  • list variables in your environment that are exported
  • kill a process
  • list all 'bash' processes running on your system in user-oriented format
  • temporarily suspend a process
  • resume a temporarily suspended process
  • look at the top processes with respect to memory usage or CPU usage
  • list all ext3 type mounted file systems
  • temporarily mount a fat file system from device sdb1 to a temporary mount point
  • bind-mount /srv/raidarray/project to /srv/nfs4/project
  • eject a cd-rom
  • power off your computer
  • reboot your computer
  • examine the exit status of the last foreground command you executed

User management / directory services

  1. date: 2010 / 05 / 03
  2. responsible: Markus
  3. topics: LDAP
  4. links for preparation:

Hostnames

Domain: course

User uid uidNumber Host name IP address VPN address
Anita anitawinkler 1013 anitawinkler.course 192.168.16.13 10.178.0.13
Benjamin 1010 benjamin.course 192.168.16.10 10.178.0.10
Fabian fabian 1012 fabian.course 192.168.16.12 10.178.0.12
Florian user1 1009 florian.course 192.168.16.9 10.178.0.9
Laszlo lkajan 1007 lkajan.course 192.168.16.7 10.178.0.7
Manfred manfred 1014 manfred.course 192.168.16.14 10.178.0.14
Markus schmidb 1000 markus.course 192.168.16.15 10.178.0.15
Lothar richter ? richter.course ? ?
Stefan seemayer 1005 seemayer.course 192.168.16.5 10.178.0.5
Kristine 1002 seitzc.course 192.168.16.2 10.178.0.2
Tatjana tanya 1006 tanya.course 192.168.16.6 10.178.0.6

Programming Challenge

  • Set up a directory service (LDAP) for the practical and define a fitting directory structure. Please use LDIF files!
  • Check the monitoring tools and log files for your installed directory service.
  • Connect your address book, e.g. in Mozilla Thunderbird with the LDAP. Check the monitoring tools and log files for your installed directory service.
  • Connect the user management of your debian installation to your LDAP

Update

In class we have reached the state where your user management is connected to the LDAP database.

  1. Read out of the above table your numeric user id and change it (also the numeric group id) in your LDAP database
  2. Change the ownership of any file owned by the old user id. You will have to update the owner and group to the new number on files most likely located in /home/<username>.

Configure your LDAP server as a replication provider so your fellow course members can pick up your user records:

  1. Read these man pages: syncrepl section in slapd.conf, slapo-syncprov
  2. Configure the syncprov overlay (I think two extra lines in your slapd.conf)

Advanced challenge

  • Secure the connection to the LDAP server with TLS or SSL and a server certificate

Hints and tips

  • Only make changes in /etc/ldap or /etc/openldap
  • You will need these packages: slapd ldap-utils migrationtools
  • Save all the original conigurations, e.g. in XXX.old
  • use ldapsearch to test your ldap server from the command line
  • You will need these packages: ldap-utils libpam-ldap libnss-ldap nscd

Laszlo's hints and tips

The goal: have regular users in the LDAP directory.

  1. apt-get install slapd libnss-ldap nscd
  2. Edit /etc/hosts, have your IP address associated with your host name (see above table), like: 192.168.16.X <NAME>.course <NAME>
  3. Edit /etc/default/slapd, have: SLAPD_SERVICES="ldapi:///"
  4. dpkg-reconfigure slapd, set domain: course; organisation: course; administrator distinguished name (DN): cn=admin,dc=course
  5. Use 'ldapvi --discover -D cn=admin,dc=course -h ldapi:///' to
    • add two organizational units ou=people,dc=course and ou=group,dc=course
    • add your group as an 'objectClass: posixGroup': cn=<usr>,ou=group,dc=course
    • add your user as an 'objectClass: posixAccount; objectClass shadowAccount; objectClass inetOrgPerson': uid=<usr>,ou=people,dc=course
    • use slappasswd to generate the encrypted form of your password
  6. Edit /etc/nsswitch.conf, have 'files ldap' for the passwd, group and shadow databases
  7. dpkg-reconfigure libnss-ldap libpam-ldap
  8. Edit /etc/pam.d/common-*, prepend:
    account sufficient pam_ldap.so ignore_unknown_user ignore_authinfo_unavail

Advanced+:

  • tinyca2

Report

by Benjamin Wellmann

Mail, DNS

  1. date: 2010 / 05 / 10 + 2010 / 05 / 17
  2. responsible: Laszlo
  3. topics: mail transfer agent, spam, MX, procmail, Maildir and mbox formats, Internet Message Access Protocol (IMAP), webmail, authentication, X.509 certificates,
  4. links for preparation:

Packages recommended for installation

  • Please install these packages without recommendations (check out 'Install recommended packages automatically' in the preferences in aptitude):
    • gnome-core
    • xorg
    • iceweasel, icedove

Programming Challenge

  • Set up and configure a DNS server
  • Set up and configure a mail server
  • Set up and IMAP server
  • Use Thunderbird / Icedove to send a mail to another course member

Hints and tips

  • packages to install: bind9, dnsutils; postfix, postfix-doc, bsd-mailx; dovecot-imapd; icedove; ca-certificates; procmail
  • Postfix configuration: choose 'Internet site'

Name server

  • Edit /etc/bind/named.conf.local, add:
zone "course" {
       type master;
       file "/etc/bind/db.course";
};

zone "16.168.192.in-addr.arpa" {
       type master;
       file "/etc/bind/db.192.168.16";
};
  • Edit /etc/bind/db.course and /etc/bind/db.192.168.16, have:

/etc/bind/db.course: 

;
; BIND reverse data file for broadcast zone
;
$TTL    86400
@       IN      SOA     lkajan.course. root.lkajan.course. (
                       10051701         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      lkajan.course.

lkajan          A       192.168.16.7
<other course members>

/etc/bind/db.192.168.16: 

;
; BIND reverse data file for broadcast zone
;
$TTL    86400
@       IN      SOA     lkajan.course. root.lkajan.course. (
                       10051701         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      lkajan.course.

7       PTR     lkajan.course.
<other course members>
  • Replace lkajan and 192.168.16.7 with your host name as appropriate
  • Test-load your named configuration: named-checkconf -z
  • Restart the name server
  • Update your /etc/resolv.conf with your own name server:
search course
nameserver 127.0.0.1
...
  • Test the name server with: host <name>.course; dig <name>.course; ping <name>.course

Mail server

  • Edit /etc/postfix/main.cf, review the mydestination line as well as mynetworks. Add your IP address to the mynetworks list.
  • Restart postfix and check that all is well with it (check the logs)
  • Add a root alias to your regular account (/etc/aliases); recreate the alias database
  • Edit your ~/.procmailrc and configure Maildir / mbox delivery as you prefer
# Maildir:
DEFAULT="$HOME/Maildir/"
  • Send a mail to yourself as root
  • Examine the mail log and check if the mail was delivered well

Dovecot (IMAP server)

  • Edit /etc/dovecot/dovecot.conf
  • Do not change anything but look at the protocols and the authentication: PAM does the work for us; ssl_cert_file and ssl_key_file: this is where we are going to secure communication to the server

Thunderbird / Icedove

  • Start Icedove and configure a new mail server:
    • Email address: <username>@<hostname>.course
    • Type: IMAP
    • Incoming server: <hostname>.course or 127.0.0.1
    • Outgoing server: <hostname>.course or 127.0.0.1
    • Configure LDAP: Preferences -> Composition -> Addressing -> Directory server -> Edit directories -> Add:
Hostname: localhost
Base DN: dc=course
Port n: 389
Bind DN: uid=<username>,,ou=people,dc=course
    • Make sure your LDAP server serves connections to ldap://localhost/ (check in /etc/default/slapd
    • Try sending a mail to another course member, e.g. Laszlo Kajan <lkajan@lkajan.course>

Advanced challenge

  • Set up procmail recipes that automatically:
    • reply to the sender that you are busy preparing to an exam if the mail subject contains the word 'work'
    • reply to the sender that you are busy with your work when the subject contains the work 'exam'
    • reply to the sender that you are ill when the subject contains both 'exam' and 'work'
  • Create a postfix regular expression table for aliases and use this table to deliver all mail matching the pattern '/^sink/' to /dev/null

Report

by Fabian Grandke

Webserver

  1. date: 2010 / 05 / 31
  2. responsible: Markus
  3. topics: Apache, CGI, PHP
  4. links for preparation:
  5. Intro slides: File:Web intro slides.pdf

Vi Exam: Monday, 31 / May / 2010

There will be a Vi exam at Monday, 31th of May, at the beginning of the practical session. The exams takes 10 minutes and you will get 25 to 30 questions about Vi. You are allowed to use the computer and Internet.

Example test questions

  • How do you start vi?
  • How do you start vi and automatically open a file at line 22?
  • What is the command to save a file?
  • How do you quit vi without saving the file?
  • What is the command to jump to line 33?
  • What is the command to junp to the middle row of your current window?
  • What is the command to delete a complete line?
  • What is the command to delete one word (including the spaces)?
  • How do you copy the a word?
  • What is the command to replace all name '/tmp' to '/var/tmp'?
  • How do you open a new file?
  • What is the command to delete the next 10 lines?
  • What is the command to move the courser to the next occurrence of 'tmp'?
  • What is the shortcut for undo?
  • Which key stoke gives you information (e.g. number of lines) about your file?

Programming Challenge

  • Install your own apache webserver and enable https (secure!).
  • Enable PHP.
  • Test different ways of access control.
  • To test your application implement a simple private homepage with PHP and only give access to people in our own network.

Hints and tips

  • Start the xserver
  • Update your system (new kernel? reboot!)

Apache

  • Install apache (package apache2). Which package have to be installed additional?
  • Test the apache at localhost and change the default content (e.g. add your name!). [/var/www]
  • Get used to the apache process manager /etc/init.d/apache2. How do you reload new configurations? At which pid is the apache process running?
  • Check the default configurations in /etc/apache2. Where goes the error log? Which modules are available?
  • Create a virtual host (/etc/apache2/sites-available/XXX) to access you website with you hostname + enable the virtual host (a2ensite)
NameVirtualHost *
<VirtualHost *>
 ServerName markus
 ServerAlias markus.course markus.course
 DocumentRoot /var/www
 ServerAdmin webmaster@markus.course
 # Logfiles:
 CustomLog /var/log/apache2/access2.log combined
 ErrorLog /var/log/apache2/error2.log
 LogLevel warn
 <Location />
   Options Indexes FollowSymLinks MultiViews
   AllowOverride None
   Order allow,deny
   allow from all
 </Location>
</VirtualHost>
  • Add an Alias and a Redirect. What is the difference?

https

First of all we need a server certificate: 

cd /etc/apache2/ssl
openssl req -new -x509 -nodes -out markus.course.crt -keyout markus.course.key
  • In "Common Name" you have to write your correct hostname.
  • We are not going to authenticate our certificate from an official organization (this costs money). We can life with the warnings and with importing the certificate at the browser start. CAcert.org is one certification unit.

We now have to change our virtual host: 

Listen 192.168.16.15:443
<VirtualHost 192.168.16.15:443>
 ServerName markus.course
 DocumentRoot /var/www
 # SSL
 SSLEngine On
 SSLCipherSuite HIGH:MEDIUM
 SSLCertificateFile    /etc/apache2/ssl/markus.course.crt
 SSLCertificateKeyFile /etc/apache2/ssl/markus.course.key

And restart the apache in the end.

Modules and PHP

There are many modules to extend the apache: 'apache2 -l', 'a2enmod'. Which modules are listed from these commands?

  • A list of all debian apache modules you can get with 'aptitude search apache2-mod-'

Install the php5 module

  • PHP is not compatible to worker-MPM. The webserver has to be changed to prefork-MPM (slower). We are using debian packages, luckily this works out of the box.
  • To test our PHP installation we use the phpinfo function
echo '<?php phpinfo() ?>'   >> /var/www/test.php
  • Which info do you get from this table?

Access Control

There are different ways to control the access.

Advanced challenge

  • enable suphp
  • access control with LDAP

Report

by Anita Winkler

Connectivity: VPN, (V)LAN, firewall, IDS, networking

  1. date: 2010 / 06 / 07
  2. responsible: Laszlo
  3. topics: on-line self defence, firewall, intrusion detection systems, firewall administration; WAN and LAN, routing, masquerading, IPsec, racoon/charon IPsec keying, strongswan, OpenVPN
  4. links for preparation:

Programming challenge with hints and tips

Arrange firewalling for your machine

  1. Install the shorewall-shell and shorewall-doc packages
  2. Copy the default configuration into /etc/shorewall from /usr/share/doc/shorewall-common/default-config/
  3. Review all files in /etc/shorewall as well as /etc/default/shorewall with special attention to: hosts, interfaces, policy, rules and zones
  4. Define an eth+ interface in interfaces with no zone or broadcast assigned
  5. Have the following zones (in addition to fw): cours for the course LAN, net for the rest of the Internet; configure the hosts file with the respective IP ranges
  6. Add policies for the zones: REJECT for net -> $FW, REJECT for cours -> $FW. Allow 'info' logging.
  7. Add rules allowing traffic from all zones to $FW for e.g. HTTP, HTTPS, IPsec, IPsecah, IPsecnat, LDAP, LDAPS, SMTP, SMTPS, Submission, SSH, Ping using macros from /usr/share/shorewall/.
  8. Check the configuration with shorewall check; start the shorewall firewall
  9. Examine the generated firewall rules with iptables-save
  10. Examine the number of packets and amount of data matched by the rules with iptables -L -vx
  11. Use nc from the netcat package to connect to another course member to send and receive messages

Connect to the course VPN server

  1. Install the tinyca package
  2. Download your certificate authority (CA) from http://rostlab.org/~lkajan/tblcourse2010
  3. Check your timezone (dpkg-reconfigure tzdata), install the ntp and ntpdate packages for automatic time synchronization, set your time now manually so that it is correct
  4. Start tinyca2 and import your CA certificate
    1. Cancel the 'Create CA' window that appears
    2. Click 'Import CA' and import the files you got in the tar archive
    3. The CA private key has .key extension - you will have to click All files, same for the Index File (serial)
    4. You will have to manually edit the field for Directory containing certificates to <yourCAdir>/certs
    5. After clicking OK double check that the CA is really for you. Your course host name should appear in the Common Name field, e.g. TBL Course 2010 anitawinkler CA
    6. This TinyCA version seems a bit broken. You will have to manually copy the serial file from the CA archive over the serial file in ~/.TinyCA/<CAname>/.
      You will get errors in the following steps (when signing the new certificate) if your serial file is not all right.
  5. Create a new client certificate with the following DN:
    1. Common Name: your course hostname, e.g. lkajan.course
    2. Organizational Unit: host
    3. Choose No to Add eMail Address to Subject DN
  6. Export the new certificate and the corresponding key. Make sure you export the key Without Passphrase.
  7. Add the Debian testing repository to your apt configuration, marking the stable release as default. We would like to use the 'testing' version of StrongSwan for VPN.
    1. Create the file /etc/apt/sources.list.d/testing.list with content
deb http://ftp.de.debian.org/debian/ testing main non-free contrib
deb-src http://ftp.de.debian.org/debian/ testing main non-free contrib

deb http://security.debian.org/ testing/updates main contrib non-free
deb-src http://security.debian.org/ testing/updates main contrib non-free
    1. Create/edit /etc/apt/apt.conf and have
APT {
  Default-Release "stable";
};

in it.

  1. Resynchronize the package index files from their sources (apt-get update)
  2. Install the 4.3.2-1.3 (testing) version of strongswan. Resolving dependencies will bring in some other testing packages, that is all right for now.
  3. Move your exported key into /etc/ipsec.d/private/
  4. Move your exported certificate into /etc/ipsec.d/certs/
  5. Create a symbolic link or copy ~/.TinyCA/<yourCA>/cacert.pem to /etc/ipsec.d/cacerts/<yourCA>.pem
  6. Download my CA certificate from http://rostlab.org/kajlaCA.crt into /etc/ipsec.d/cacerts/
  7. Configure strongswan with your certificate and key you exported above:
    1. Edit /etc/ipsec.secrets: comment out the include ... line and have
: RSA <your_exported_key_filename_in_/etc/ipsec.d/private/>
    1. Edit /etc/ipsec.conf and have
conn i12r-studfilesrv
       left=131.159.28.107
       leftid="C=DE, ST=Bayern, L=Munich, O=i12, OU=strongswan, CN=i12r-studfilesrv.informatik.tu-muenchen.de"
       leftcert=strongswan@i12r-studfilesrv-cert.pem
       leftsubnet=10.178.0.0/24
       right=%defaultroute
       rightcert=<your_exported_cert_filename_in_/etc/ipsec.d/certs/>
       rightsourceip=<your_VPN_IP_from_above_table_e.g._10.178.0.7>
       keyexchange=ikev2
       auto=start
  1. Restart the ipsec service (/etc/init.d/ipsec restart). Watch and read /var/log/syslog (as root) for messages from the charon daemon: this will report all errors or success of the connection initiation.
  2. Use setkey -DP (from ipsec-tools package) to list the Security Policy Database (SPD). You should see your VPN IP address in the list
  3. If another course member is using the VPN you can now try to ping/nc him/her

Advanced challenge

  1. Install version 3.1.2-3.1 of ganglia-monitor and configure it to monitor hosts on the vpn. Cluster name: 'tbl2010', owner: 'course'; udp_send_channel/ttl=2. Open port 8649 tcp and udp in your firewall.
  2. use gstat -a to list all hosts present on the VPN

Report

by Manfred Roos

Databases and SQL

  1. date: 2010 / 06 / 14
  2. responsible: Markus
  3. topics: DBMS, mysql, postgreSql, SQL
  4. links for preparation:
  5. Intro slides: File:Mysql slides.pdf

Programming Challenge

  • Install and configure a mysql server.
    • There should be a database with your name and a user with your name.
    • Only the user should have access to the database from localhost.
  • Create a backup from your database.
  • Install phpMyAdmin to provide a nice frontend for users.
  • Create you own database and fill a table with some data. For example use use PHP or perl!

Hints and Tips

mysql

Install the package 'mysql-server'

  • which additional packages will be installed? Which client?
  • you have to set a mysql-root password during the installation process

A nice GUI based mysql client is the 'mysql-navigator'

mysql administration

At this point, we create basic permissions for a user and database: Allow access to localhost to all databases, and a computer which is also on the network - e.g "laszlo.course"

Connect to your MySQL server: 

mysql -u root -p

To exit the mysql server print 

mysql> quit;

To access the user, host databases, etc... type this; 

mysql> use mysql;

To give localhost permission to access all databases, enter this: 

mysql> insert into 
        -> host(host,db,Select_priv, Insert_priv, Update_priv, 
        -> Delete_priv, Create_priv, Drop_priv)
        -> values('localhost','%','Y','Y','Y','Y','Y','Y');
  • the '%' can be replaced with a database name. The '%' is a wildcard.

To allow access from another hostname (in this case "lkajan.course") add this: 

mysql> insert into 
 -> host(host,db,Select_priv, Insert_priv, Update_priv, 
 -> Delete_priv, Create_priv, Drop_priv)
 -> values('lkajan.course','%','Y','Y','Y','Y','Y','Y');

To create a user 'markus' who can access the MySQL server from localhost, type this: 

mysql> insert into 
        -> user (host, user, password)
        -> values('localhost','markus',password('XXX'));

To give the user access from another hostname, domain, etc... add other entries accordingly. For example, to give user Laszlo access from lkajan.course : 

mysql> insert into 
        -> user (host, user, password)
        -> values('lkajan.course','Laszlo',password('XXX'));

To give the user permissions to access a database from localhost, add this entry and change with your appropriate information: 

mysql> insert into
     -> db (host,db,user,Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv)
     -> values ('localhost','mydatabase','markus','Y','Y','Y','Y','Y','Y');

Finally, create the actual database (in this case, 'mydatabase') type this: 

mysqladmin -u root -p create mydatabase
  • After prompting you for a password, it should create the database.
  • At this point, you must reload MySQL. Type:
mysqladmin -u root -p reload

Much more you can find at http://dev.mysql.com/doc/refman/5.0/en/index.html

  • Now, create a database with your name and a user with your name. Only the user should have access to the database from localhost.

Backup

Use 'mysqldump' to create a backup from your databases.

phpMyAdmin

Install the package 'phpmyadmin'

  • you have to choose a webserver during the installation process (apache2)
  • you have to add this new website to your apache configuration
    • /etc/sites-enabled/YOURVIRTUALHOST add 
      

      Include /etc/phpmyadmin/apache.conf
      

      • 

    • What is included in /etc/phpmyadmin/apache.conf
      • 

    • Do not forgett to reload the apache
    • 

  • go to http://localhost/phpmyadmin and login
    • 

  • now we can do everything (?) with the webinterface

    • can you do all the changes we did with the command line?


Writing and Reading


We are going to use perl to write into the database.



  • Install 'perl' and a module to connect to a database ('libdbi-perl')
    • 

  • Some pseudo perl code


use DBI;
my @con = ('DBI:mysql:cgdiso:localhost', 'markus', 'XXX'); 
my $dbh = DBI->connect( @con ) || die "Database connection not made: $DBI::errstr";
# create table
$sql = qq{
 CREATE TABLE disorder (
 Id VARCHAR(12) NOT NULL,
 Residue LONGTEXT NOT NULL,
 MD LONGTEXT NOT NULL,
 time DATETIME NOT NULL,
 PRIMARY KEY id (Id)
) TYPE = MyISAM COMMENT ='md results';
};
$sth = $dbh->prepare( $sql );
sth->execute();
$sth->finish();
# CLOSE connection
$dbh->disconnect();



  • Can you do the same with any other programming language?


Advanced Challenge


  • Install and configure PostgreSQL
    • 

  • Install a nice user front-end
    • 

  • Use your preferred programming language to access the database


Report


by Tatyana Goldberg




Default web applications


  1. date: 2010 / 06 / 21
    2. 

  2. responsible: Markus
    3. 

  3. topics: CMS, wiki, Bug tracking, forum, calendar
    4. 

  4. links for preparation:

    5. 

  5. Intro slides: File:Web service slides.pdf


Programming Challenge - REVIEWED


  • Install a wiki implementation.
    • 

  • Install the CMS of your choice (e.g. Typo3).
    • 

  • Connect the user management of the CMS with your LDAP.
    • 

  • Create a simple web page with your CMS for the practical.


Programming challenge


Please send us screenshots (including the URL of the browser) from 



  • one page in your wiki.
    • 

  • The front- and backend from your CMS and
    • 

  • 2 pictures from different subpages in your CMS.


Hints and Tips


wiki


There are many different wiki engines:




A good and stable debian version is the package 'mediawiki' (http://www.mediawiki.org)



  • add to your virtual host file


Include /etc/mediawiki/apache.conf 



  • Which domain do you now use to access the mediawiki?
    • 

  • Follow the instructions and install the wiki.

    • Discuss with your neighbor a suitable configuration.
      • 

    • What permissions do you set for the 'LocalSettings.php' file?
      • 

    • At least, add a logo and only allow registered users to change the content ($wgGroupPermissions).


CMS


There are many different CMS implementations.
A good and stable debian version is the package 'typo3' (www.typo3.org)



  • add to your virtual host file


Include /etc/typo3-dummy/apache.conf 



  • Which domain do you now use to access the typo?
    • 

  • To get into the installation process you have to create an empty file (ENABLE_INSTALL_TOOL) in the /var/lib/typo3-dummy/typo3conf directory.
    • 

  • Follow the installation instructions!

    • Discuss a suitable configuration with your neighbor.
      • 

    • You have to create a new db user!
      • 

    • Do not remove the typo3/install folder and the ENABLE_INSTALL_TOOL file. Do it after the complete configuration!
    • 

  • Login into typo3, make the finale configuration and build a simple web page!

    • Which URL do you use to access the front-end and which for the back-end?
      • 

    • To connect the back-end to the LDAP you need an extension 'eu_ldap'. For the configuration see the manual (at the typo3.org page)
      • 

    • Create a simple page object with some simple content objects.
      • 

    • Try to build up a page tree structure. If you choose a good layout, you can recover the page tree structure in the navigation.
      • 

    • Now you need a Layout Template. You can use 'TemplaVoila' and build your own one. Or google the web for some example templates to include them.
      • 

    • After several configuration you have to clean the CACHE to see the changes! (top right)
    • 

  • Add some nice extensions (you can find them at typo3.org):

    • realurl: for nicer URLs
      • 

    • tt_news: to present news in a nice layout
      • 

    • ...
    • 

  • Now you can remove the typo3/install folder and the ENABLE_INSTALL_TOOL file


CMS Advanced Challenge


  • As layout for your website rebuild the TUM Corporate Design and find a way to export it from your installation (to import into another system).
    • 

  • Integrate an internal user section (connection to the LDAP and use https) for the front-end!. There is an (several) extension available for Typo3.


Wiki Advanced Challenge



Report


by Tatjana Goldberg



  • no presentation
    • 

  • xxx.pdf


Computer Clusters and External Services


  1. date: 2010 / 06 / 28
    2. 

  2. responsible: Markus
    3. 

  3. topics: computer cluster hardware, batch systems
    4. 

  4. links for preparation:



Programming Challenge


We will get a guided tour in the LRZ to see all the server and services in live operation.



Hints and Tips


  • We meet at 2 p.m. in the rostlab!
    • 

  • Please be in time.
    • 

  • A photo identification is required.
    • 

  • There is air condition in the server rooms. Please dress appropriate.


Modules in programming, development and design


  1. date: 2010 / 07 / 05
    2. 

  2. responsible: Laszlo
    3. 

  3. topics: monolithic programming vs. modular programming in practice; case analysis
    4. 

  4. links for preparation:

    • real-life programming examples will be provided for students to analyse and discuss


Programming Challenge


  • analyse provided example case
    • 

  • propose solution to lower difficulty of maintenance and deployment
    • 

  • implement solution


Hints and Tips


Advanced Challenge


Report


Report to be prepared together with following session (creating a Debian package).

No presentation.



Package a piece of software, create a Debian or RPM package


  1. date: 2010 / 07 / 12
    2. 

  2. responsible: Laszlo
    3. 

  3. topics: successful packaging practices, .tar.gz, packaging for Debian/Ubuntu, packaging for RPM-based distributions
    4. 

  4. links for preparation:



Programming Challenge


  • package the assignment from the previous practical


Hints and Tips


  1. Prepare a distributable archive of your software (make distcheck, you will be using automake and autoconf I expect). Your archive should be named like this: <packagename>-<sersion>.tar.gz
    2. 

  2. Untar the archive to a fresh location
    3. 

  3. cd into dir created above
    4. 

  4. dh_make -e <email@addre.ss> -c gpl --createorig # man dh_make. Is your package gpl? Give a reasonable maintainer email address. Answer questions: you are most likely creating a single binary type of package.
    5. 

  5. cd debian
    6. 

  6. Remove all unnecessary files from this directory. You will most likely not need any of the .ex and .EX files there.
    7. 

  7. You will most likely not need dirs, docs, README.Debian
    8. 

  8. Edit the remaining files except compat: changelog control copyright rules

    1. Make sure you have a gpg key (man gpg) on your keyring for the signature you use in the changelog, e.g. Laszlo Aan <lkan@rolab.gr>.  This gpg key will be used to sign packages you build.  I have a gpg key:

      • $ gpg -k # uid                  Laszlo Aan <lkan@rolab.gr>
        • 

      • If you do not yet have the key create it now: $ gpg --gen-key # follow instructions. No need to add a comment.
      2. 

    2. In control define a section, say science. Define build dependencies (Build-Depends) and dependencies (Depends) as needed. Give a home page or remove the line. Change Architecture to all in case your package contains only architecture independent files (such as perl scripts and modules). Give a Description and a longer description below.
      3. 

    3. Edit copyright - fill in the parts as appropriate, delete the parts you do not need
      4. 

    4. Edit rules and add --sysconfdir=/etc to the ./configure line
    9. 

  9. cd ..
    10. 

  10. Execute dpkg-buildpackage (from dpkg-dev package)
    11. 

  11. Exec debc - see what you packaged and where
    12. 

  12. cd ..
    13. 

  13. Your binary package is <packagename>_<version>_<arch>.deb. The other files .changes, .diff.gz, .dsc and .orig.tar.gz make up the source package. You can now try to install the Debian package on a Debian or Ubuntu machine: dpkg --install <packagename>_<version>_<arch>.deb .
    14. 

  14. If you want to keep the debian subdirectory in your repository just copy it into your working copy and put it under version control. This will allow you to run dpkg-buildpackage directly from your working copy.


Check out the svn-buildpackage package for a solution how to integrate version control and Debian package building.

Use any Debian or Ubuntu package for ideas and examples: apt-get source <your_favourite_package>. Look into the debian subdirectory of it.



Report and presentation


no one



Version control, diff, patching


SKIPPED



RAID, file systems, network file systems


SKIPPED









Retrieved from "https://rostlab.org/owiki/index.php?title=The_bioinformatics_lab_2010&oldid=1250"