The bioinformatics lab 2010
This practical is a hands-on training that will make you successful in a Bioinformatic lab! All the services and details from bottom up, e.g. Wiki, version control, compute cluster, software packaging
More details can be found at the http://rostlab.informatik.tu-muenchen.de/cms/biolab2010/ website.
Contents
- 1 External media / installing a Linux OS / Debian stable / Linux kernel
- 2 Shell scripts, command line (terminal) editors and compilation
- 3 User management / directory services
- 4 Mail, DNS
- 5 Webserver
- 6 Connectivity: VPN, (V)LAN, firewall, IDS, networking
- 7 Databases and SQL
- 8 Default web applications
- 9 Computer Clusters and External Services
- 10 Modules in programming, development and design
- 11 Package a piece of software, create a Debian or RPM package
- 12 Version control, diff, patching
- 13 RAID, file systems, network file systems
External media / installing a Linux OS / Debian stable / Linux kernel
- date: 2010 / 04 / 19
- responsible: Laszlo
- topics: Linux distributions, rpm vs. deb, stable vs. bleeding edge; decisions to make at the time of system installation. Partitioning, LVM, cryptsetup; kernel modules, boot loader, initramdisk; dpkg, apt, aptitude, apticron; sysctl, /proc, /sys
- links for preparation:
- google.com: popularity of Linux
- http://distrowatch.com/
- http://debian.org/ http://ubuntu.com/ http://redhat.com/ etc.
- http://kernel.org/
- http://www.gnu.org/
- http://www.gnu.org/software/grub/
Programming challenge
Install and configure a Debian stable base system on your USB stick. Be careful with the partition, only use the USB device. Do not forget to enable boot on USB in your BIOS. Get familiar with vim.
Hints and tips
- Choose English so Laszlo can help
- Choose manual partitioning and make sure you leave alone all drives except the USB stick
- Note the device name of the USB stick, something like 'sda', 'sdb'...
- Have 3 partitions on the USB stick: one for a /boot file system which should be bootable and not more that 512 MB; one for swap space - 512MB; one for a / (root) file system. ext3 is a good choice for the /boot and / file systems.
- There is no proxy for accessing the internet
- Choose NOT to install the boot loader into the first hard drive: 'Install the GRUB boot loader to the master boot record?' NO
- Install the boot loader to the device you noted at step 3 above, e.g. /dev/sdb. Be careful not to install GRUB onto your hard drive.
- Boot from the USB key. Chances are this is going go fail because the way the installer installs GRUB (the boot loader) is not the best for removable devices.
- If booting fails with Error 17: Can not mount selected partition - the error we observed so far - hit 'e' on the grub screen to edit the first menu option. Hit 'e' again to edit the line root (hdX,Y). 'X' represents the hard drives as the BIOS sees them, numbered from 0. Error 17 indicates the the presently set hard drive does not contain a /boot partition where indicated by the 'root' line, so experiment with other numbers, e.g. 0:
root (hd0,0)
. Do not change the second number 'Y': that indicates the partition and is likely to be correct. Try booting with the new 'root' line (hit 'b'). This modification is not permanent. - Getting the 'root' line right should enable you to boot into Linux. The permanent solution is to use GRUB2 instead of GRUB. GRUB2 uses the universally unique identifier (UUID) to indicate root and boot partitions and so can handle removable devices well.
- Once you booted Linux log in as root and replace GRUB with GRUB2. You are going to need a working network connection for this step (courtesy of Fabian). Start the package manager 'aptitude'. Hit '?' and learn the keys' meanings. Search for the package 'grub2' and mark it for installation. Install it. Answer NO to the question whether to chain-load GRUB2. When the installation is complete exit aptitude.
- Install this version of grub to the master boot record of the USB key: execute
update-grub
. Find the current device name of the usb key, e.g. withmount
, looking for a line like /dev/sda3 on /boot type ext3. In this case the USB stick is /dev/sda. Executegrub-install
with the device for the USB key, e.g.grub-install /dev/sda
. - Reboot from the USB key - this time it should boot cleanly into Linux.
Advanced challenge
Install Debian into an encrypted partition.
Materials and methods
Report
By Stefan Seemayer:
Shell scripts, command line (terminal) editors and compilation
- date: 2010 / 04 / 26
- responsible: Markus
- topics: vim, emacs, shell scripts, gcc, makefiles, automake, autoconf, kernel and module compilation
- links for preparation:
Programming challenge
Write a short C / perl / python program that reads text from a file, removes all spaces and writes the result back into a file. Create a distributable tar ball of your program using automake and autoconf. Add a man page as well. Get familiar with terminal-based text editors. We recommend you implement this programming challenge using vim.
Advanced challenge
Compile the kernel of your choice making your hard drive, video card and network card drivers compiled in (not in modules). Do the kernel compilation on an in-memory file system (such as tmpfs) - doing it on the USB stick would take forever.
Hints and tips
Automake and autoconf
- Edit your package sources list (/etc/apt/sources.list) and enable the 'contrib' and 'non-free' sections of the repository: add contrib and non-free after 'main' on each deb and deb-src line. Refresh the package cache.
- Install the 'make', 'make-doc', 'automake' and 'autoconf-doc' packages: these provide automake, autoconf and the info documentation
- Learn to navigate the info browser (do
info automake
, press '?' and read) - Read section 1 Introduction and 2 Autotools Introduction up to and including 2.2.4 Standard Configuration Variables.
- Follow the examples (e.g. 'zardoz') in the automake info to create your Makefile.am and configure.ac. You will want to have at least these macros in your configure.ac:AC_INIT
AM_INIT_AUTOMAKE
AC_CONFIG_FILES
AC_OUTPUT- Use the documentation to find out more about these.
- You can use the
--prefix
./configure option to test the install target at a custom location (e.g.--prefix=/tmp/test
) - I recommend you use the pod syntax to create the man page. Install the 'perl-doc' package to gain access to the 'perlpod' manpage. Read:
man perlpod; man pod2man
. - Create rules in Makefile.am to have make generate the manpage for your script from a .pod source
- If your program is a script, use the SCRIPTS primary instead of PROGRAMS
- Use the DATA primary to account for the .pod source and the MANS primary for the man page
- Make sure the .pod source is not installed but the generated man page is (use the automake 'dist' and 'noinst' prefixes as appropriate).
- Make sure your package passes the
make distcheck
test
Kernel
- Mount (permanently if you want) a tmpfs on /tmp: 'man mount; man fstab'. Compiling a kernel with IO to the USB stick would take forever. You should have enough memory to hold the entire kernel compilation in memory. A 2GB tmpfs will be enough if you un-check the
Kernel hacking -> Kernel debugging
option (1166 MB was used in my case) - I advise you use make menuconfig to configure the kernel of your choice
- You can use the
lspci -v
command to find out about the driver of your network interface. Use this page http://www.debian.org/distrib/packages to find out what package provides lspci - Remember to create an initramdisk for your compiled kernel in case you wish to test it.
man update-initramfs
Materials and methods
- File:Linuxbasics 10.pdf from Lothar Richter
Report
By Florian Seifert:
Linux Command Exam: Monday, 10 / May / 2010
There will be a Linux Command Exam at Monday, 10th of May, at the beginning of the practical session. The exams takes 10 minutes and you will get 25 to 30 questions about Linux command lines. You are allowed to use the computer and Internet.
Example test questions
What command would you use to:
- remove an empty directory
- remove a potentially filled directory
- remove all files with '.pl~' extension in a directory tree
- switch the group write permission on on all files that match the '*.pl' extension in a directory tree
- list a directory with files sorted on modification time in reverse (newest on bottom)
- copy a directory tree to another location in an 'archiving' way
- copy a directory tree to another computer in an efficient way
- create the directory /tmp/test/src/linux with one command when only /tmp exists
- open a man page file in your present directory that is not within the regular man path
- print your PATH? What is the function of the PATH environment variable?
- add your present working directory to your path
- look at the contents of a text file (name at least two tools)
- compare two text files
- list your environment
- list variables in your environment that are exported
- kill a process
- list all 'bash' processes running on your system in user-oriented format
- temporarily suspend a process
- resume a temporarily suspended process
- look at the top processes with respect to memory usage or CPU usage
- list all ext3 type mounted file systems
- temporarily mount a fat file system from device sdb1 to a temporary mount point
- bind-mount /srv/raidarray/project to /srv/nfs4/project
- eject a cd-rom
- power off your computer
- reboot your computer
- examine the exit status of the last foreground command you executed
User management / directory services
- date: 2010 / 05 / 03
- responsible: Markus
- topics: LDAP
- links for preparation:
- http://en.wikipedia.org/wiki/Directory_service
- http://www.openldap.org/
- http://www.linux-magazin.de/Heft-Abo/Ausgaben/2001/05/Straffe-Verwaltung (in German)
- http://www.zytrax.com/books/ldap/ch2/
- http://docstore.mik.ua/orelly/perl/sysadmin/appb_01.htm
- http://quark.humbug.org.au/publications/ldap/system_auth/sage-au/system_auth.html
- http://www.skills-1st.co.uk/papers/ldap-schema-design-feb-2005/ldap-schema-design-feb-2005.html
Hostnames
Domain: course
User | uid | uidNumber | Host name | IP address | VPN address |
---|---|---|---|---|---|
Anita | anitawinkler | 1013 | anitawinkler.course | 192.168.16.13 | 10.178.0.13 |
Benjamin | 1010 | benjamin.course | 192.168.16.10 | 10.178.0.10 | |
Fabian | fabian | 1012 | fabian.course | 192.168.16.12 | 10.178.0.12 |
Florian | user1 | 1009 | florian.course | 192.168.16.9 | 10.178.0.9 |
Laszlo | lkajan | 1007 | lkajan.course | 192.168.16.7 | 10.178.0.7 |
Manfred | manfred | 1014 | manfred.course | 192.168.16.14 | 10.178.0.14 |
Markus | schmidb | 1000 | markus.course | 192.168.16.15 | 10.178.0.15 |
Lothar | richter | ? | richter.course | ? | ? |
Stefan | seemayer | 1005 | seemayer.course | 192.168.16.5 | 10.178.0.5 |
Kristine | 1002 | seitzc.course | 192.168.16.2 | 10.178.0.2 | |
Tatjana | tanya | 1006 | tanya.course | 192.168.16.6 | 10.178.0.6 |
Programming Challenge
- Set up a directory service (LDAP) for the practical and define a fitting directory structure. Please use LDIF files!
- Check the monitoring tools and log files for your installed directory service.
- Connect your address book, e.g. in Mozilla Thunderbird with the LDAP. Check the monitoring tools and log files for your installed directory service.
- Connect the user management of your debian installation to your LDAP
Update
In class we have reached the state where your user management is connected to the LDAP database.
- Read out of the above table your numeric user id and change it (also the numeric group id) in your LDAP database
- Change the ownership of any file owned by the old user id. You will have to update the owner and group to the new number on files most likely located in /home/<username>.
Configure your LDAP server as a replication provider so your fellow course members can pick up your user records:
- Read these man pages: syncrepl section in slapd.conf, slapo-syncprov
- Configure the syncprov overlay (I think two extra lines in your slapd.conf)
Advanced challenge
- Secure the connection to the LDAP server with TLS or SSL and a server certificate
Hints and tips
- Only make changes in /etc/ldap or /etc/openldap
- You will need these packages: slapd ldap-utils migrationtools
- Save all the original conigurations, e.g. in XXX.old
- use ldapsearch to test your ldap server from the command line
- You will need these packages: ldap-utils libpam-ldap libnss-ldap nscd
Laszlo's hints and tips
The goal: have regular users in the LDAP directory.
- apt-get install slapd libnss-ldap nscd
- Edit /etc/hosts, have your IP address associated with your host name (see above table), like: 192.168.16.X <NAME>.course <NAME>
- Edit /etc/default/slapd, have: SLAPD_SERVICES="ldapi:///"
- dpkg-reconfigure slapd, set domain: course; organisation: course; administrator distinguished name (DN): cn=admin,dc=course
- Use 'ldapvi --discover -D cn=admin,dc=course -h ldapi:///' to
- add two organizational units ou=people,dc=course and ou=group,dc=course
- add your group as an 'objectClass: posixGroup': cn=<usr>,ou=group,dc=course
- add your user as an 'objectClass: posixAccount; objectClass shadowAccount; objectClass inetOrgPerson': uid=<usr>,ou=people,dc=course
- use slappasswd to generate the encrypted form of your password
- Edit /etc/nsswitch.conf, have 'files ldap' for the passwd, group and shadow databases
- dpkg-reconfigure libnss-ldap libpam-ldap
- Edit /etc/pam.d/common-*, prepend:
- account sufficient pam_ldap.so ignore_unknown_user ignore_authinfo_unavail
Advanced+:
- tinyca2
Report
by Benjamin Wellmann
- Slides: File:Slides ldap wellmann.pdf
- Protocol: File:Protocol ldap wellmann2.pdf
Mail, DNS
- date: 2010 / 05 / 10 + 2010 / 05 / 17
- responsible: Laszlo
- topics: mail transfer agent, spam, MX, procmail, Maildir and mbox formats, Internet Message Access Protocol (IMAP), webmail, authentication, X.509 certificates,
- links for preparation:
- google.com: comparison of mail transfer agents (MTA) for Linux
- http://www.postfix.org/
- http://www.exim.org/
- http://spamassassin.apache.org/
- http://www.procmail.org/
- http://www.dovecot.org/
- http://squirrelmail.org/
- Simple Authentication and Security Layer (SASL)
Packages recommended for installation
- Please install these packages without recommendations (check out 'Install recommended packages automatically' in the preferences in aptitude):
- gnome-core
- xorg
- iceweasel, icedove
Programming Challenge
- Set up and configure a DNS server
- Set up and configure a mail server
- Set up and IMAP server
- Use Thunderbird / Icedove to send a mail to another course member
Hints and tips
- packages to install: bind9, dnsutils; postfix, postfix-doc, bsd-mailx; dovecot-imapd; icedove; ca-certificates; procmail
- Postfix configuration: choose 'Internet site'
Name server
- Edit /etc/bind/named.conf.local, add:
zone "course" { type master; file "/etc/bind/db.course"; }; zone "16.168.192.in-addr.arpa" { type master; file "/etc/bind/db.192.168.16"; };
- Edit /etc/bind/db.course and /etc/bind/db.192.168.16, have:
/etc/bind/db.course:
; ; BIND reverse data file for broadcast zone ; $TTL 86400 @ IN SOA lkajan.course. root.lkajan.course. ( 10051701 ; Serial 604800 ; Refresh 86400 ; Retry 2419200 ; Expire 86400 ) ; Negative Cache TTL ; @ IN NS lkajan.course. lkajan A 192.168.16.7 <other course members>
/etc/bind/db.192.168.16:
; ; BIND reverse data file for broadcast zone ; $TTL 86400 @ IN SOA lkajan.course. root.lkajan.course. ( 10051701 ; Serial 604800 ; Refresh 86400 ; Retry 2419200 ; Expire 86400 ) ; Negative Cache TTL ; @ IN NS lkajan.course. 7 PTR lkajan.course. <other course members>
- Replace lkajan and 192.168.16.7 with your host name as appropriate
- Test-load your named configuration: named-checkconf -z
- Restart the name server
- Update your /etc/resolv.conf with your own name server:
search course nameserver 127.0.0.1 ...
- Test the name server with: host <name>.course; dig <name>.course; ping <name>.course
Mail server
- Edit /etc/postfix/main.cf, review the mydestination line as well as mynetworks. Add your IP address to the mynetworks list.
- Restart postfix and check that all is well with it (check the logs)
- Add a root alias to your regular account (/etc/aliases); recreate the alias database
- Edit your ~/.procmailrc and configure Maildir / mbox delivery as you prefer
# Maildir: DEFAULT="$HOME/Maildir/"
- Send a mail to yourself as root
- Examine the mail log and check if the mail was delivered well
Dovecot (IMAP server)
- Edit /etc/dovecot/dovecot.conf
- Do not change anything but look at the protocols and the authentication: PAM does the work for us; ssl_cert_file and ssl_key_file: this is where we are going to secure communication to the server
Thunderbird / Icedove
- Start Icedove and configure a new mail server:
- Email address: <username>@<hostname>.course
- Type: IMAP
- Incoming server: <hostname>.course or 127.0.0.1
- Outgoing server: <hostname>.course or 127.0.0.1
- Configure LDAP: Preferences -> Composition -> Addressing -> Directory server -> Edit directories -> Add:
Hostname: localhost Base DN: dc=course Port n: 389 Bind DN: uid=<username>,,ou=people,dc=course
- Make sure your LDAP server serves connections to ldap://localhost/ (check in /etc/default/slapd
- Try sending a mail to another course member, e.g. Laszlo Kajan <lkajan@lkajan.course>
Advanced challenge
- Set up procmail recipes that automatically:
- reply to the sender that you are busy preparing to an exam if the mail subject contains the word 'work'
- reply to the sender that you are busy with your work when the subject contains the work 'exam'
- reply to the sender that you are ill when the subject contains both 'exam' and 'work'
- Create a postfix regular expression table for aliases and use this table to deliver all mail matching the pattern '/^sink/' to /dev/null
Report
by Fabian Grandke
Webserver
- date: 2010 / 05 / 31
- responsible: Markus
- topics: Apache, CGI, PHP
- links for preparation:
- Intro slides: File:Web intro slides.pdf
Vi Exam: Monday, 31 / May / 2010
There will be a Vi exam at Monday, 31th of May, at the beginning of the practical session. The exams takes 10 minutes and you will get 25 to 30 questions about Vi. You are allowed to use the computer and Internet.
Example test questions
- How do you start vi?
- How do you start vi and automatically open a file at line 22?
- What is the command to save a file?
- How do you quit vi without saving the file?
- What is the command to jump to line 33?
- What is the command to junp to the middle row of your current window?
- What is the command to delete a complete line?
- What is the command to delete one word (including the spaces)?
- How do you copy the a word?
- What is the command to replace all name '/tmp' to '/var/tmp'?
- How do you open a new file?
- What is the command to delete the next 10 lines?
- What is the command to move the courser to the next occurrence of 'tmp'?
- What is the shortcut for undo?
- Which key stoke gives you information (e.g. number of lines) about your file?
Programming Challenge
- Install your own apache webserver and enable https (secure!).
- Enable PHP.
- Test different ways of access control.
- To test your application implement a simple private homepage with PHP and only give access to people in our own network.
Hints and tips
- Start the xserver
- Update your system (new kernel? reboot!)
Apache
- Install apache (package apache2). Which package have to be installed additional?
- Test the apache at localhost and change the default content (e.g. add your name!). [/var/www]
- Get used to the apache process manager /etc/init.d/apache2. How do you reload new configurations? At which pid is the apache process running?
- Check the default configurations in /etc/apache2. Where goes the error log? Which modules are available?
- Create a virtual host (/etc/apache2/sites-available/XXX) to access you website with you hostname + enable the virtual host (a2ensite)
NameVirtualHost * <VirtualHost *> ServerName markus ServerAlias markus.course markus.course DocumentRoot /var/www ServerAdmin webmaster@markus.course # Logfiles: CustomLog /var/log/apache2/access2.log combined ErrorLog /var/log/apache2/error2.log LogLevel warn <Location /> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Location> </VirtualHost>
- Add an Alias and a Redirect. What is the difference?
https
First of all we need a server certificate:
cd /etc/apache2/ssl openssl req -new -x509 -nodes -out markus.course.crt -keyout markus.course.key
- In "Common Name" you have to write your correct hostname.
- We are not going to authenticate our certificate from an official organization (this costs money). We can life with the warnings and with importing the certificate at the browser start. CAcert.org is one certification unit.
We now have to change our virtual host:
Listen 192.168.16.15:443 <VirtualHost 192.168.16.15:443> ServerName markus.course DocumentRoot /var/www # SSL SSLEngine On SSLCipherSuite HIGH:MEDIUM SSLCertificateFile /etc/apache2/ssl/markus.course.crt SSLCertificateKeyFile /etc/apache2/ssl/markus.course.key
And restart the apache in the end.
Modules and PHP
There are many modules to extend the apache: 'apache2 -l', 'a2enmod'. Which modules are listed from these commands?
- A list of all debian apache modules you can get with 'aptitude search apache2-mod-'
Install the php5 module
- PHP is not compatible to worker-MPM. The webserver has to be changed to prefork-MPM (slower). We are using debian packages, luckily this works out of the box.
- To test our PHP installation we use the phpinfo function
echo '<?php phpinfo() ?>' >> /var/www/test.php
- Which info do you get from this table?
Access Control
There are different ways to control the access.
- With IP-Address:
- change your virtual host file
- # Single IP-adress:
- Allow from 192.168.16.5 192.168.16.15
- # IP-network:
- Allow from 192.168.16.
- Allow from 10.1.0.0/16
- Allow from 10.2.0.0/255.255.0.0
- # all computer in a DNS-Domain:
- Allow from informatik.tu-muenchen.de
- Documentation: http://httpd.apache.org/docs/2.0/mod/mod_access.html
- change your virtual host file
- With Password:
- change the from-statement in your virtual host file:
- AuthType Basic
- AuthName "internal"
- AuthUserFile /etc/apache2/htpasswd
- Require valid-user
- create passwd file
- htpasswd -c /etc/apache2/htpasswd Username
- How do you remove user from passwd?
- How do you define valid users?
- Documentation:
- change the from-statement in your virtual host file:
- You can enable access control with certificates or LDAP too!
Advanced challenge
- enable suphp
- access control with LDAP
Report
by Anita Winkler
Connectivity: VPN, (V)LAN, firewall, IDS, networking
- date: 2010 / 06 / 07
- responsible: Laszlo
- topics: on-line self defence, firewall, intrusion detection systems, firewall administration; WAN and LAN, routing, masquerading, IPsec, racoon/charon IPsec keying, strongswan, OpenVPN
- links for preparation:
- google.com: cyber attacks
- http://www.shorewall.net/
- http://www.snort.org/
- http://www.strongswan.org/
- http://openvpn.net/
Programming challenge with hints and tips
Arrange firewalling for your machine
- Install the shorewall-shell and shorewall-doc packages
- Copy the default configuration into /etc/shorewall from /usr/share/doc/shorewall-common/default-config/
- Review all files in /etc/shorewall as well as /etc/default/shorewall with special attention to: hosts, interfaces, policy, rules and zones
- Define an eth+ interface in interfaces with no zone or broadcast assigned
- Have the following zones (in addition to fw): cours for the course LAN, net for the rest of the Internet; configure the hosts file with the respective IP ranges
- Add policies for the zones: REJECT for net -> $FW, REJECT for cours -> $FW. Allow 'info' logging.
- Add rules allowing traffic from all zones to $FW for e.g. HTTP, HTTPS, IPsec, IPsecah, IPsecnat, LDAP, LDAPS, SMTP, SMTPS, Submission, SSH, Ping using macros from /usr/share/shorewall/.
- Check the configuration with shorewall check; start the shorewall firewall
- Examine the generated firewall rules with iptables-save
- Examine the number of packets and amount of data matched by the rules with iptables -L -vx
- Use nc from the netcat package to connect to another course member to send and receive messages
Connect to the course VPN server
- Install the tinyca package
- Download your certificate authority (CA) from http://rostlab.org/~lkajan/tblcourse2010
- Check your timezone (dpkg-reconfigure tzdata), install the ntp and ntpdate packages for automatic time synchronization, set your time now manually so that it is correct
- Start tinyca2 and import your CA certificate
- Cancel the 'Create CA' window that appears
- Click 'Import CA' and import the files you got in the tar archive
- The CA private key has .key extension - you will have to click All files, same for the Index File (serial)
- You will have to manually edit the field for Directory containing certificates to <yourCAdir>/certs
- After clicking OK double check that the CA is really for you. Your course host name should appear in the Common Name field, e.g. TBL Course 2010 anitawinkler CA
- This TinyCA version seems a bit broken. You will have to manually copy the serial file from the CA archive over the serial file in ~/.TinyCA/<CAname>/.
- You will get errors in the following steps (when signing the new certificate) if your serial file is not all right.
- Create a new client certificate with the following DN:
- Common Name: your course hostname, e.g. lkajan.course
- Organizational Unit: host
- Choose No to Add eMail Address to Subject DN
- Export the new certificate and the corresponding key. Make sure you export the key Without Passphrase.
- Add the Debian testing repository to your apt configuration, marking the stable release as default. We would like to use the 'testing' version of StrongSwan for VPN.
- Create the file /etc/apt/sources.list.d/testing.list with content
deb http://ftp.de.debian.org/debian/ testing main non-free contrib deb-src http://ftp.de.debian.org/debian/ testing main non-free contrib deb http://security.debian.org/ testing/updates main contrib non-free deb-src http://security.debian.org/ testing/updates main contrib non-free
- Create/edit /etc/apt/apt.conf and have
APT { Default-Release "stable"; };
in it.
- Resynchronize the package index files from their sources (apt-get update)
- Install the 4.3.2-1.3 (testing) version of strongswan. Resolving dependencies will bring in some other testing packages, that is all right for now.
- Move your exported key into /etc/ipsec.d/private/
- Move your exported certificate into /etc/ipsec.d/certs/
- Create a symbolic link or copy ~/.TinyCA/<yourCA>/cacert.pem to /etc/ipsec.d/cacerts/<yourCA>.pem
- Download my CA certificate from http://rostlab.org/kajlaCA.crt into /etc/ipsec.d/cacerts/
- Configure strongswan with your certificate and key you exported above:
- Edit /etc/ipsec.secrets: comment out the include ... line and have
: RSA <your_exported_key_filename_in_/etc/ipsec.d/private/>
- Edit /etc/ipsec.conf and have
conn i12r-studfilesrv left=131.159.28.107 leftid="C=DE, ST=Bayern, L=Munich, O=i12, OU=strongswan, CN=i12r-studfilesrv.informatik.tu-muenchen.de" leftcert=strongswan@i12r-studfilesrv-cert.pem leftsubnet=10.178.0.0/24 right=%defaultroute rightcert=<your_exported_cert_filename_in_/etc/ipsec.d/certs/> rightsourceip=<your_VPN_IP_from_above_table_e.g._10.178.0.7> keyexchange=ikev2 auto=start
- Restart the ipsec service (/etc/init.d/ipsec restart). Watch and read /var/log/syslog (as root) for messages from the charon daemon: this will report all errors or success of the connection initiation.
- Use setkey -DP (from ipsec-tools package) to list the Security Policy Database (SPD). You should see your VPN IP address in the list
- If another course member is using the VPN you can now try to ping/nc him/her
Advanced challenge
- Install version 3.1.2-3.1 of ganglia-monitor and configure it to monitor hosts on the vpn. Cluster name: 'tbl2010', owner: 'course'; udp_send_channel/ttl=2. Open port 8649 tcp and udp in your firewall.
- use gstat -a to list all hosts present on the VPN
Report
by Manfred Roos
- File:Roos slides.pdf
- protocol missing
Databases and SQL
- date: 2010 / 06 / 14
- responsible: Markus
- topics: DBMS, mysql, postgreSql, SQL
- links for preparation:
- Intro slides: File:Mysql slides.pdf
Programming Challenge
- Install and configure a mysql server.
- There should be a database with your name and a user with your name.
- Only the user should have access to the database from localhost.
- Create a backup from your database.
- Install phpMyAdmin to provide a nice frontend for users.
- Create you own database and fill a table with some data. For example use use PHP or perl!
Hints and Tips
mysql
Install the package 'mysql-server'
- which additional packages will be installed? Which client?
- you have to set a mysql-root password during the installation process
A nice GUI based mysql client is the 'mysql-navigator'
mysql administration
At this point, we create basic permissions for a user and database: Allow access to localhost to all databases, and a computer which is also on the network - e.g "laszlo.course"
Connect to your MySQL server:
mysql -u root -p
To exit the mysql server print
mysql> quit;
To access the user, host databases, etc... type this;
mysql> use mysql;
To give localhost permission to access all databases, enter this:
mysql> insert into -> host(host,db,Select_priv, Insert_priv, Update_priv, -> Delete_priv, Create_priv, Drop_priv) -> values('localhost','%','Y','Y','Y','Y','Y','Y');
- the '%' can be replaced with a database name. The '%' is a wildcard.
To allow access from another hostname (in this case "lkajan.course") add this:
mysql> insert into -> host(host,db,Select_priv, Insert_priv, Update_priv, -> Delete_priv, Create_priv, Drop_priv) -> values('lkajan.course','%','Y','Y','Y','Y','Y','Y');
To create a user 'markus' who can access the MySQL server from localhost, type this:
mysql> insert into -> user (host, user, password) -> values('localhost','markus',password('XXX'));
To give the user access from another hostname, domain, etc... add other entries accordingly. For example, to give user Laszlo access from lkajan.course :
mysql> insert into -> user (host, user, password) -> values('lkajan.course','Laszlo',password('XXX'));
To give the user permissions to access a database from localhost, add this entry and change with your appropriate information:
mysql> insert into -> db (host,db,user,Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv) -> values ('localhost','mydatabase','markus','Y','Y','Y','Y','Y','Y');
Finally, create the actual database (in this case, 'mydatabase') type this:
mysqladmin -u root -p create mydatabase
- After prompting you for a password, it should create the database.
- At this point, you must reload MySQL. Type:
mysqladmin -u root -p reload
Much more you can find at http://dev.mysql.com/doc/refman/5.0/en/index.html
- Now, create a database with your name and a user with your name. Only the user should have access to the database from localhost.
Backup
Use 'mysqldump' to create a backup from your databases.
phpMyAdmin
Install the package 'phpmyadmin'
- you have to choose a webserver during the installation process (apache2)
- you have to add this new website to your apache configuration
- /etc/sites-enabled/YOURVIRTUALHOST add
- Include /etc/phpmyadmin/apache.conf
- What is included in /etc/phpmyadmin/apache.conf
- Do not forgett to reload the apache
- /etc/sites-enabled/YOURVIRTUALHOST add
- go to http://localhost/phpmyadmin and login
- now we can do everything (?) with the webinterface
- can you do all the changes we did with the command line?
Writing and Reading
We are going to use perl to write into the database.
- Install 'perl' and a module to connect to a database ('libdbi-perl')
- Some pseudo perl code
use DBI; my @con = ('DBI:mysql:cgdiso:localhost', 'markus', 'XXX'); my $dbh = DBI->connect( @con ) || die "Database connection not made: $DBI::errstr"; # create table $sql = qq{ CREATE TABLE disorder ( Id VARCHAR(12) NOT NULL, Residue LONGTEXT NOT NULL, MD LONGTEXT NOT NULL, time DATETIME NOT NULL, PRIMARY KEY id (Id) ) TYPE = MyISAM COMMENT ='md results'; }; $sth = $dbh->prepare( $sql ); sth->execute(); $sth->finish(); # CLOSE connection $dbh->disconnect();
- Can you do the same with any other programming language?
Advanced Challenge
- Install and configure PostgreSQL
- Install a nice user front-end
- Use your preferred programming language to access the database
Report
by Tatyana Goldberg
Default web applications
- date: 2010 / 06 / 21
- responsible: Markus
- topics: CMS, wiki, Bug tracking, forum, calendar
- links for preparation:
- Intro slides: File:Web service slides.pdf
Programming Challenge - REVIEWED
- Install a wiki implementation.
- Install the CMS of your choice (e.g. Typo3).
- Connect the user management of the CMS with your LDAP.
- Create a simple web page with your CMS for the practical.
Programming challenge
Please send us screenshots (including the URL of the browser) from
- one page in your wiki.
- The front- and backend from your CMS and
- 2 pictures from different subpages in your CMS.
Hints and Tips
wiki
There are many different wiki engines:
A good and stable debian version is the package 'mediawiki' (http://www.mediawiki.org)
- add to your virtual host file
Include /etc/mediawiki/apache.conf
- Which domain do you now use to access the mediawiki?
- Follow the instructions and install the wiki.
- Discuss with your neighbor a suitable configuration.
- What permissions do you set for the 'LocalSettings.php' file?
- At least, add a logo and only allow registered users to change the content ($wgGroupPermissions).
CMS
There are many different CMS implementations. A good and stable debian version is the package 'typo3' (www.typo3.org)
- add to your virtual host file
Include /etc/typo3-dummy/apache.conf
- Which domain do you now use to access the typo?
- To get into the installation process you have to create an empty file (ENABLE_INSTALL_TOOL) in the /var/lib/typo3-dummy/typo3conf directory.
- Follow the installation instructions!
- Discuss a suitable configuration with your neighbor.
- You have to create a new db user!
- Do not remove the typo3/install folder and the ENABLE_INSTALL_TOOL file. Do it after the complete configuration!
- Login into typo3, make the finale configuration and build a simple web page!
- Which URL do you use to access the front-end and which for the back-end?
- To connect the back-end to the LDAP you need an extension 'eu_ldap'. For the configuration see the manual (at the typo3.org page)
- Create a simple page object with some simple content objects.
- Try to build up a page tree structure. If you choose a good layout, you can recover the page tree structure in the navigation.
- Now you need a Layout Template. You can use 'TemplaVoila' and build your own one. Or google the web for some example templates to include them.
- After several configuration you have to clean the CACHE to see the changes! (top right)
- Add some nice extensions (you can find them at typo3.org):
- realurl: for nicer URLs
- tt_news: to present news in a nice layout
- ...
- Now you can remove the typo3/install folder and the ENABLE_INSTALL_TOOL file
CMS Advanced Challenge
- As layout for your website rebuild the TUM Corporate Design and find a way to export it from your installation (to import into another system).
- Integrate an internal user section (connection to the LDAP and use https) for the front-end!. There is an (several) extension available for Typo3.
Wiki Advanced Challenge
- Enable LDAP Authentication for your MediaWiki (debian package: mediawiki-extensions-ldapauth)
- Try any other debian stable wiki implementation. There are several: http://packages.debian.org/search?suite=default§ion=all&arch=any&searchon=names&keywords=wiki
Report
by Tatjana Goldberg
- no presentation
- xxx.pdf
Computer Clusters and External Services
- date: 2010 / 06 / 28
- responsible: Markus
- topics: computer cluster hardware, batch systems
- links for preparation:
- http://en.wikipedia.org/wiki/Batch-queuing_system
- http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.53.1154&rep=rep1&type=pdf, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.109.3233&rep=rep1&type=pdf
- http://gridengine.sunsource.net/, https://computing.llnl.gov/linux/slurm/
- http://www.lrz-muenchen.de/
Programming Challenge
We will get a guided tour in the LRZ to see all the server and services in live operation.
Hints and Tips
- We meet at 2 p.m. in the rostlab!
- Please be in time.
- A photo identification is required.
- There is air condition in the server rooms. Please dress appropriate.
Modules in programming, development and design
- date: 2010 / 07 / 05
- responsible: Laszlo
- topics: monolithic programming vs. modular programming in practice; case analysis
- links for preparation:
- real-life programming examples will be provided for students to analyse and discuss
Programming Challenge
- analyse provided example case
- propose solution to lower difficulty of maintenance and deployment
- implement solution
Hints and Tips
Advanced Challenge
Report
Report to be prepared together with following session (creating a Debian package).
No presentation.
Package a piece of software, create a Debian or RPM package
- date: 2010 / 07 / 12
- responsible: Laszlo
- topics: successful packaging practices, .tar.gz, packaging for Debian/Ubuntu, packaging for RPM-based distributions
- links for preparation:
Programming Challenge
- package the assignment from the previous practical
Hints and Tips
- Prepare a distributable archive of your software (make distcheck, you will be using automake and autoconf I expect). Your archive should be named like this: <packagename>-<sersion>.tar.gz
- Untar the archive to a fresh location
- cd into dir created above
- dh_make -e <email@addre.ss> -c gpl --createorig # man dh_make. Is your package gpl? Give a reasonable maintainer email address. Answer questions: you are most likely creating a single binary type of package.
- cd debian
- Remove all unnecessary files from this directory. You will most likely not need any of the .ex and .EX files there.
- You will most likely not need dirs, docs, README.Debian
- Edit the remaining files except compat: changelog control copyright rules
- Make sure you have a gpg key (man gpg) on your keyring for the signature you use in the changelog, e.g. Laszlo Aan <lkan@rolab.gr>. This gpg key will be used to sign packages you build. I have a gpg key:
- $ gpg -k # uid Laszlo Aan <lkan@rolab.gr>
- If you do not yet have the key create it now: $ gpg --gen-key # follow instructions. No need to add a comment.
- In control define a section, say science. Define build dependencies (Build-Depends) and dependencies (Depends) as needed. Give a home page or remove the line. Change Architecture to all in case your package contains only architecture independent files (such as perl scripts and modules). Give a Description and a longer description below.
- Edit copyright - fill in the parts as appropriate, delete the parts you do not need
- Edit rules and add --sysconfdir=/etc to the ./configure line
- Make sure you have a gpg key (man gpg) on your keyring for the signature you use in the changelog, e.g. Laszlo Aan <lkan@rolab.gr>. This gpg key will be used to sign packages you build. I have a gpg key:
- cd ..
- Execute dpkg-buildpackage (from dpkg-dev package)
- Exec debc - see what you packaged and where
- cd ..
- Your binary package is <packagename>_<version>_<arch>.deb. The other files .changes, .diff.gz, .dsc and .orig.tar.gz make up the source package. You can now try to install the Debian package on a Debian or Ubuntu machine: dpkg --install <packagename>_<version>_<arch>.deb .
- If you want to keep the debian subdirectory in your repository just copy it into your working copy and put it under version control. This will allow you to run dpkg-buildpackage directly from your working copy.
Check out the svn-buildpackage package for a solution how to integrate version control and Debian package building.
Use any Debian or Ubuntu package for ideas and examples: apt-get source <your_favourite_package>. Look into the debian subdirectory of it.
Report and presentation
no one
Version control, diff, patching
SKIPPED
RAID, file systems, network file systems
SKIPPED