The bioinformatics lab SS 2012

From Rost Lab Open
Jump to: navigation, search

This practical is a hands-on training that will make you successful in a Bioinformatics lab! This term we focus on packaging software for Debian/Ubuntu.

Materials from previous courses are available at The bioinformatics lab SS 2011 and The bioinformatics lab. There you can find many hints and tips and the protocols and presentation slides from last year. You may use them, but especially the protocols and presentations you have to prepare with your own content and own style!


Contents

Installing Linux / Into a Virtual Machine / Debian stable

  • Date: 2012 / 04 / 17
  • Tutor: Laszlo Kajan
  • Topics: Linux distributions, rpm vs. deb, stable vs. bleeding edge; decisions to make at the time of system installation; disk partitioning, LVM, RAID, iSCSI; choice of file system; deb package management; virtualization basics.

Questions

  • What is Linux?
  • What are the leading Linux distributions?
  • What is free software?
  • What are the Debian releases?
  • What is in a (server) computer in terms of hardware?
  • How do you define specifications for new hardware? - server roles and vendor lock-in.
  • Where can you boot a (server) computer from for installing an operating system?
  • What happens when you install an operating system?
  • How to distribute 24 disks for 6 persons working on 6 projects?

Slides

Links for preparation

Programming challenge

  • Install and configure a Debian stable base system in a virtual machine.
    1. Create a maximum 10G image for the virtual machine.
    2. Generate a random MAC address for the virtual machine.
    3. Use 'kvm' for virtualization with no more than 1.5GB of RAM.
    4. Connect to the VDE virtual network with these kvm arguments: -net vde,sock=/var/run/kvm0.ctl -net nic,macaddr=$MAC.
    5. Install Debian Stable on the virtual machine from the installation CD image at /home/tbl2012/tbl2012/debian-6.0.4-amd64-netinst.iso. For faster installation, do not select to install a 'Desktop Environment' but do select the 'SSH Server'.
    6. Configure the virtual machine with a static IP address and host name according to the table below. Netmask: 255.255.255.0 , gateway: 192.168.16.1, name server: 192.168.16.1, domain: tbl, search: tbl.
    7. Change the user and group ID of yourself (in your virtual machine) to the numbers in the table below.
    8. Start your virtual machine in the background and connect/reconnect to it via VNC and SSH.
  • Get familiar with vim.

Hints and tips

  1. Choose English so Laszlo can help.
  2. kvm-img is convenient for creating the image - virtual disk of your virtual machine.
  3. A MAC address is of the form '%02X:%02X:%02X:%02X:%02X:%02X'. Everyone needs a unique address on our virtual network.
  4. man kvm - read this man page. kvm starts the hypervisor, a process that runs your virtual machine. kvm is the virtual hardware your virtual machine runs on. The parameters on the kvm command line describe what that virtual hardware should be like. More about virtualization later (session 3).
  5. The 'virtio' interface for disks and network interfaces is recommended.
  6. 'writeback' caching is recommended for virtual disks.
  7. Configure a USB tablet device for the virtual machine so that you can use the mouse pointer easily.
  8. In case you have a Mac, you may encounter key mapping issues. Thanks to Christian, the solution is to use the -k kvm argument with de if appropriate. The way to define a libvirt domain with this parameter is to have this in the XML of the domain: <qemu:commandline><qemu:arg value='-k'/><qemu:arg value='de'/></qemu:commandline>.
  9. When running your virtual machine in the background, use a VNC display to see the VGA display of your VM. VNC is also much more suitable than an X window for not-very-fast network connections.
  10. You may want to give the VNC session a password: (-vnc :0,password - password literally, replace 0 with your number (last to digits) in the table below). You will have to set the password in the hypervisor monitor. In order to access the hypervisor monitor, redirect it to a UNIX socket: -chardev socket,id=monitor,path=$HOME/debian-stable.monitor,server,nowait -mon chardev=monitor,mode=readline. The hypervisor monitor allows you to change parameters of kvm while it is active. Connect to the UNIX socket of the hypervisor monitor with: nc -U $HOME/debian-stable.monitor. Now you can use change vnc password in the hypervisor monitor to set the VNC password (just type it into that nc command you started).
  11. In order to connect to the VNC virtual display, you have to forward a port of your laptop to i12r-tbl, for example like this: ssh -L 5902:127.0.0.1:5902 i12r-tbl. Then use a VNC client such as 'xvnc4viewer' to connect, like this: vncviewer :2.

When the initial installation is complete:

  1. Edit the kvm command line and remove the arguments that 'attach' the CD-rom install image to the virtual machine - you are not going to need this any more.
  2. You have to indicate that your file system image file (the 10GB file) is bootable. If you use -drive to specify the virtual disk (instead of say -hda), add boot=on to the parameters of the -drive argument or you get No bootable device..
  3. In case you see name resolution problems (e.g. Could not resolve 'ftp.de.debian.org'), you have to edit /etc/resolv.conf and set the name server, plus optionally the domain and search into it (as given above): use man resolv.conf to learn how to put these into that file.
  4. Choose English as the default language of your system - use 'dpkg-reconfigure locales', add 'en_US.UTF-8' to the list of locales and make it the default.
  5. You can change keyboard layout in the terminal with 'loadkeys'.
  6. Install the 'vim-nox' package, call the 'vimtutor' command and start learning vim UNLESS you are proficient with emacs. We are soon going to write a vim test!

Advanced challenge

Get a desktop environment installed in your virtual machine. Watch the space: 10G may not be enough for installing all the recommended packages and games, etc.

Presentation

Host names

Domain: tbl

User uid uidNumber/gidNumber Host name IP address L2 virtual machine IP address
Laszlo lkajan 1002 lkajan.tbl 192.168.16.2 192.168.16.32
Ariane ariane 1003 ariane.tbl 192.168.16.3 192.168.16.33
Benjamin benjamin 1004 benjamin.tbl 192.168.16.4 192.168.16.34
Carina carina 1005 carina.tbl 192.168.16.5 192.168.16.35
Cedric cedric 1006 cedric.tbl 192.168.16.6 192.168.16.36
Christian christian 1007 christian.tbl 192.168.16.7 192.168.16.37
Christof christof 1008 christof.tbl 192.168.16.8 192.168.16.38
Daniel daniel 1009 daniel.tbl 192.168.16.9 192.168.16.39
Diana diana 1010 diana.tbl 192.168.16.10 192.168.16.40
Eva eva 1011 eva.tbl 192.168.16.11 192.168.16.41
Jens jens 1012 jens.tbl 192.168.16.12 192.168.16.42
Julia julia 1013 julia.tbl 192.168.16.13 192.168.16.43
Yannick yannick 1014 yannick.tbl 192.168.16.14 192.168.16.44
Simon simon 1015 simon.tbl 192.168.16.15 192.168.16.45

Terminal-based text editors, tar.gz packaging

  • Date: 2012 / 04 / 24
  • Tutor: Lothar Richter, Laszlo Kajan
  • Topics: text editors, shell scripts, preparing distributions
  • Questions
    • What is a 'tarball' package?
    • What does 'make' do?
    • Can 'make' make use of multiple cores/CPUS for parallel builds?
    • What are standard 'make' targets?
    • What is a 'staged installation'?
    • What does 'architecture dependent/independent' mean?
    • How to prepare software for distribution:
      • Compiled code (C/C++)?
      • Perl and Python?
      • Java?
    • What is Plain Old Documentation (POD) format?

Programming challenge

Write a short program that reads text from a file, removes all spaces and writes the result back into a file. Document the modules and executables with man/info pages (POD format recommended). Create a distributable tar ball of your solution (code, examples and documentation).

  • Use either autotools to package the compiled version or
  • the appropriate mechanism for perl or python for the script version.
    • For Perl packaging Module::Build or ExtUtils::MakeMaker are appropriate.
    • If you solve the challenge with python, be ready to present your solution to the group and Laszlo: he is not a python programmer but would like to know how it is done.
  • Make sure that:
    • If you use autotools, your package complies with the GNU standard: do not use the 'foreign' option. This is only a recommendation in this term, I should have set this as a requirement earlier.
    • The 'make distcheck' command succeeds.
  • Staged installation works (in both cases).

make challenge

This challenge is to be solved by creating a file Makefile that is processed by the make command. There is no need for autotools here.

  • Have your full name in file full.name.

Create a Makefile that:

  • Is executable and is processed by make like make -f Makefile [ARGS]. (hint: have a #! line on the top and use the -f make argument).
    E.g. './Makefile' should invoke the 'all' target, './Makefile clean' should invoke the 'clean' target.
  • Has rule(s) to create files first.name and last.name from the full name (hint: man cut).
    E.g. './Makefile first.name' should produce the expected name in file first.name.
  • Has rule(s) to create files *.xxd with the hex dump of all *.name files (hint: man xxd).
    E.g. './Makefile last.xxd' should produce the hex dump of last.name in file last.xxd.
  • Has rule(s) to create checksum files *.chk for all *.name files (hint: man sha1sum).
    E.g. './Makefile full.chk' should produce the checksum of full.name in file full.chk.
  • Has a clean target to remove all generated files.
  • Has a default target named all that prints out the filename and respective contents of all *.xxd and *.chk files.
  • Has a help target that lists the available targets upon invocation (hint: use the .PHONY target).
  • Take advantage of 'pattern rules' and variables, assigned with 'substitution references' when possible.

For the curious among you: if you run make with -j4 and insert some sleep statements and run top or ps or pstree, you can observe the parallel make processes that are spawn.

When ready with the challenges, send Laszlo the completed tar.gz packages (the result of make dist/make distcheck or equivalent) and the solution of the make challenge (the Makefile).

Get familiar with terminal-based text editors. We recommend you implement this programming challenge using vim.

Hints and tips

Makefile
autotools
  1. Edit your package sources list (/etc/apt/sources.list) and enable the 'contrib' and 'non-free' sections of the repository: add contrib and non-free after 'main' on each deb and deb-src line. Refresh the package cache.
  2. Install the 'make', 'make-doc', 'automake' and 'autoconf-doc' packages: these provide automake, autoconf and the info documentation.
  3. Learn to navigate the info browser (do info automake, press '?' and read)
  4. Read section 1 Introduction and 2 Autotools Introduction up to and including 2.2.4 Standard Configuration Variables.
  5. Follow the examples (e.g. 'zardoz') in the automake info to create your Makefile.am and configure.ac. You will want to have at least these macros in your configure.ac:
    AC_INIT
    AM_INIT_AUTOMAKE
    AC_CONFIG_FILES
    AC_OUTPUT
    Use the documentation to find out more about these.
  6. You can use the --prefix ./configure option to test the install target at a custom location (e.g. --prefix=/tmp/test) - very useful for non-root testing of the installation (you would normally not test as root!).
  7. I recommend you use the pod syntax to create the man page. Install the 'perl-doc' package to gain access to the 'perlpod' manpage. Read: man perlpod; man pod2man.
  8. Create rules in Makefile.am to have make generate the manpage for your script from a .pod source
  9. If your program also contains scripts, use the SCRIPTS primary in addition to PROGRAMS.
  10. Use the DATA primary to distribute the .pod source and the MANS primary to install the man page.
  11. Make sure the .pod source is not installed but the generated man page is (use the automake 'dist' and 'noinst' prefixes as appropriate).
  12. Make sure your package passes the make distcheck test.
Perl

While it is possible to package Perl with autotools, it is not convenient, especially when it comes to setting module installation paths. Use one of Perl's ways to package perl code:

  • Prefer Module::Build to ExtUtils::MakeMaker: man Module::Build.
  • You can use '--install_base' (with Module::Build) to change the base/root directory of the installation - useful for non-root installations and testing. If you use ExtUtils::MakeMaker, use PREFIX.

Note for the future: it is not difficult to have a Module::Build/ExtUtils::MakeMaker solution inside a bigger autotools package. This way you can take advantage of the strengths of both build systems.

Advanced challenge

Package both a compiled (e.g. C, C++, Fortran) and a script (e.g. perl, python) solution of the challenge.

Advanced+ challenge

Express functionality of the compiled and script solutions in a library, module. Use Doxygen to generate documentation for your C/C++ work. Implement the basic challenge in Java, document it, prepare a distributable package of your solution.

Presentation

Hypervisors, virtualization API, cloud services

  • Date: 2012 / 05 / 08
  • Tutor: Laszlo Kajan
  • Topics: x86 hardware virtualization, QEMU/KVM, virtualization API libvirt, cloud services overview

Questions

  • What is emulation, what is virtualization?
  • What is QEMU, what is KVM?
  • What hardware can KVM virtualize?
  • What is level 0, level 1 and level 2 in nested virtualization?
  • What is the role of the 'virtio' disk interface, 'virtio' network interface model type and 'vmware' VGA card type?
  • What disk image types and formats are usable with KVM? What are their advantages and disadvantages?
  • What is libvirt? What does domain mean in the context of libvirt?
  • What does it mean to migrate a virtual machine?
  • What are the requirements for migrating a virtual machine?

Slides

Links for preparation

Programming Challenge

Familiarize yourself with libvirt. Unfortunately the libvirt documentation is quite scary. Fortunately they do have very useful examples hidden among the scary bits - use them. Also there is 'virt-manager', a GUI front end, that hides the scary bits! You can perform the libvirt part of this challenge entirely using 'virt-manager' if you have a graphical interface installed, saving you from all the scary bits.

  • Define a domain [3][4] with bridge-to-LAN [5] networking and
  • install a (small) system into it that you can ssh into.
  • Use the 'L2 virtual machine IP address' column in the table to assign a static IP address for this virtual machine.
  • Send me the domain definition of your L2 virtual machine and keep it up long enough for me to be able to ping it.
  • If you plan to do the advanced challenge, read that challenge before starting to solve this challenge.

Hints and Tips

  • Start with creating a bridge interface (say br0) in your (level 1, L1) virtual machine:
    • Don't do this in an ssh session but rather in a VNC console as a mistake may cut your connection - you will be reconfiguring the network connection.
    • Install the bridge-utils package, then man bridge-utils-interfaces and follow the first example EXCEPT for bridge_ports: instead of all, give the name of your network interface (most likely eth0, you can find out by looking for the interface name in your /etc/network/interfaces).
    • Use the address, netmask asf from the ethX interface to configure the bridge and disable the automatic configuration of the ethX interface (comment out the auto ethX line - it will become a port of the bridge). Make sure br0 is up (ifup br0) before you continue.
  • I recommend you solve this challenge using virt-manager, the GUI tool. For this you will need a running graphical interface. You can follow Julia's slides, or here is how I got mine going:
    • Add kvm arguments -vga vmware to the call if you do not have it yet and restart the (L1) virtual machine.
    • Do not attempt nested virtualization (with the -enable-nesting argument): it does not work reliably (for me).
    • Install packages gnome-core and xorg. Then running a simple startx in a VNC session should get you into the graphical interface.
  • Install packages qemu and qemu-kvm (before the next step).
  • Install packages libvirt-bin and virt-manager.
  • Download a Debian netinst image into your L1 virtual machine.
  • Put yourself into the kvm, libvirt and vde2-net groups (use usermod or edit /etc/groups and /etc/gshadow).
  • Start virt-manager (in the graphical interface) and click the icon Create new virtual machine and follow the steps:
    • The name should be <yourname>-vm, e.g. lkajan-vm.
    • Choose Local install media and use the netinst image you downloaded.
    • OS type and Version does not really matter.
    • 512M RAM is well enough.
    • Choose to create a disk image on the computer's hard drive, but make it small: 1GB should be enough.
    • In Advanced options choose to Specify shared device name and give your bridge device: br0.
    • Set a new and unique MAC address, choose architecture x86_64.
    • Click Finish: the Debian installer is booted and you can install a system as before, just much more slowly - fortunately you can leave your VNC session and reconnect to it. Install an SSH server (and standard tools) into your L2 virtual machine, but no more.
  • virt-manager has defined the domain for you. You can look at it like this (in your L1 virtual machine): virsh dumpxml <yourname>-vm. Also check out View->Details in the virt-manager menu: you can add and remove hardware to your virtual machine, you can connect and disconnect a CD ROM image.

Advanced Challenge

Now this will be fun but complex:

  • Successfully migrate an L2 virtual machine from your L1 virtual machine (acting as host) to a friend's L1 and back.

See hints and tips below:

  • Think how to provide the same environment for the L2 virtual machine on both your and your friends host:
    • Use iSCSI to provide the volume for disk device of the L2 virtual machine:
      • Set up an iSCSI target and initiator in your L1 VM and have an initiator also in your friend's (connected to the same iSCSI target). 1GB should be enough for this volume, but I would not use a loopback device in the L1 for this because it may be very slow. Instead create another (fast, so raw) 1G device in the H0 (so the tbl) machine and configure this as a second disk to your L1 virtual machine, then make this second disk the iSCSI target volume.
    • Packages for iSCSI: iscsitarget and iscsitarget-dkms for the target, open-iscsi for the initiator.
    • Configure the same bridge interface on both L1 hosts, say br0.
    • Enable the libvirt daemon to listen on tcp for both L1 hosts, check:
      • /etc/libvirt/libvirtd.conf: listen_tcp, set auth_tcp = "none" for simplicity (we trust everyone on our LAN)
      • /etc/default/libvirt-bin: libvirtd_opts
  • Proceed to install the L2 virtual machine into the iSCSI volume either with virt-manager or manually, you may find debootstrap (from debootstrap package) useful in the latter case.
  • When your L2 VM is up and your friend's L1 is configured (network, storage) to receive the virtual machine, attempt the migration. This is truly spectacular when performed from a virt-manager that is connected to both L1 hosts: use qemu+tcp://<friend>.tbl/system to access his/her machine (or qemu+ssh, but then you will need root access to the other machine). Good luck with this!

Presentation

Linux Command Exam

There will be a Linux Command Exam today, at the beginning of the practical session. The exam takes 10 minutes and you will get 25 to 30 questions about Linux command lines. You are allowed to use the computer and Internet, but not each other's help.

Example Test Questions

What command would you use to:

  • remove an empty directory
  • remove a potentially filled directory
  • remove all files with '.pl~' extension in a directory tree
  • switch the group write permission on on all files that match the '*.pl' extension in a directory tree
  • list a directory with files sorted on modification time in reverse (newest on bottom)
  • copy a directory tree to another location in an 'archiving' way
  • copy a directory tree to another computer in an efficient way, supposing some of the files are already present on the remote system
  • create the directory /tmp/test/src/linux with one command when only /tmp exists
  • open a man page file in your present directory that is not within the regular man path
  • print your PATH environment variable; what is the function of the PATH environment variable?
  • add your present working directory to your shell search path
  • look at the contents of a text file (name at least two tools)
  • compare two text files
  • list your environment
  • list variables in your environment that are exported
  • kill a process
  • list all 'bash' processes running on your system in user-oriented format
  • temporarily suspend a process
  • resume a temporarily suspended process
  • background a suspended process
  • look at the top processes with respect to memory usage or CPU usage
  • list all ext4 type mounted file systems
  • temporarily mount a fat file system from device sdb1 to a temporary mount point
  • bind-mount /srv/raidarray/project to /srv/nfs4/project
  • eject a cd-rom
  • power off your computer
  • reboot your computer
  • examine the exit status of the last foreground command you executed
  • suspend and resume a process not attached to your terminal

User management / directory services

  • Date: 2012 / 05 / 15
  • Tutor: Laszlo Kajan
  • Topics: Lightweight Directory Access Protocol (LDAP), slapd server, user authentication and (user) name services with an LDAP database

Questions

  • What are the most important system databases: '/etc/passwd', '/etc/shadow', '/etc/group' and '/etc/hosts'? Explain the information in these.
  • What are name services, such as NIS or DNS (briefly)?
  • What are Pluggable Authentication Modules (PAM) for Linux (briefly)?
  • What is LDAP (briefly)? What is the LDAP directory structure, what is the LDIF format?
  • Can you use LDAP to provide name services? - can you use PAM in conjunction with LDAP to authenticate users (man pam_ldap)?
    • Why is LDAP good for providing name services?
    • What is the directory structure like, what is in the directory when LDAP is used to provides name services and authentication via PAM? Show the common classes and attributes used for this.
  • Can you configure the LDAP server daemon 'slapd' at runtime? (man slapd-config)
  • How do you control access to your LDAP database? Explain a few example access rules:
    olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth manage by * break
    olcAccess: {1}to attrs=userPassword,shadowLastChange by self write by anonymous auth by dn="cn=admin,dc=tbl" write by * none
    olcAccess: {2}to dn.base="" by * read
    olcAccess: {3}to * by self write by dn="cn=admin,dc=tbl" write by * read

Slides

Links for preparation

Programming Challenge

  1. Set up a directory service (LDAP) for the practical and define a fitting directory structure: install 'slapd' and 'ldapvi'. Use dc=tbl as the distinguished name of the search base.
  2. Check the functioning of your installed directory service:
    • ldapvi -h ldapi:/// -b cn=config -Y EXTERNAL - this command displays (in an editor) the current configuration of the slapd (LDAP) daemon.
    • ldapvi -h ldapi:/// --discover -D cn=admin,dc=tbl - this command displays the contents of the LDAP database you have.
  3. Allow access to your LDAP database olcDatabase={1}hdb,cn=config via external authentication for gidNumber=0+uidNumber=0 by copying the corresponding olcAccess line from the configuration of the cn=config database. Root now should be able to edit the database without password (ldapvi -h ldapi:/// --discover -Y EXTERNAL).
  4. Connect the user management of your debian installation to your LDAP:
    1. Move your user and group entries from the files 'passwd', 'shadow', 'group' and 'gshadow' to the LDAP database. Use the organizationalUnit(OU) class to structure the data (user records, group records) in your database. The OU for storing user records is usually called 'people', the OU for groups is 'group'. Use the 'uid' attribute in the distinguished name(DN) of a user record (e.g. uid=lkajan); use the 'cn' attribute in the DN on a group record (e.g. cn=lkajan). Use ldapvi -h ldapi:/// -b cn=config -Y EXTERNAL (as above) to access the configuration of your database: use the class definitions of organizationalUnit, posixAccount, shadowAccount and posixGroup to know what attributes are available.
    2. Install 'libnss-ldap' (and recommended packages). When asked for the LDAP account for root, look up and give the DN of the 'admin' account in your LDAP database.
    3. Modify /etc/nsswitch.conf to use the LDAP database. Use getent to verify the functioning of the name service.
    4. Use 'pam-auth-update' or 'dpkg-reconfigure libpam-runtime' to enable LDAP authentication.
  5. Configure your LDAP server as a replication provider for database {1}. You will need the 'syncprov' overlay (man slapd-config, search for OVERLAYS, man slapo-syncprov) from the 'syncprov' module (to be loaded with 'olcModuleLoad'). You can not change the list of modules with ldapvi, you have to edit the configuration in /etc/ldap/slapd.d/cn=config/cn=module{0}.ldif. Use class 'olcSyncProvConfig' for the configuration of the overlay. Add the olcServerID attribute to cn=config, set it to your offset in the host names table.
  6. Learn to understand LDAP access control statements (olcAccess, man slapd.access(5)). Interpret the olcAccess statements that come with the default configuration.

Hints and Tips

  1. Edit /etc/hosts, have your IP address associated with your host name (see above table), like: 192.168.16.X .tbl.
  2. Add two organizational units ou=people,dc=tbl and ou=group,dc=tbl.
  3. Add your group as an 'objectClass: posixGroup': cn=<usr>,ou=group,dc=tbl.
  4. Add your user as an 'objectClass: posixAccount; objectClass shadowAccount; objectClass inetOrgPerson': uid=<usr>,ou=people,dc=tbl.
  5. Use 'slappasswd' to generate the encrypted form of your password OR use the {CRYPT} qualifier and just copy the encrypted password from /etc/shadow.
  6. It is not usually necessary, but it may be a good idea to invalidate the name service cache kept by the 'nscd' daemon after modifying user and group attributes: nscd -i passwd; nscd -i group. During testing you can altogether stop the nscd daemon.
  7. Edit /etc/nsswitch.conf, append 'ldap' to the passwd, group and shadow databases' lines.
  8. Use 'pam-auth-update' to enable LDAP for PAM (authentication, etc.).
  9. Use 'getent passwd' and 'getent group' to verify that you LDAP connection to name services works. If you still have your user and group defined in /etc/passwd and /etc/group (and their shadow) files, you should see your user and group entry listed twice by the getent commands. The second is the one that comes from your LDAP server since 'ldap' appears second (to 'files' or 'compat') in /etc/nsswitch.conf.
  10. If you see that getent returns the right records for yourself and your group, remove your user and group entry from /etc/{passwd,shadow,group,gshadow}. Try to log into your virtual machine as yourself to test that it works.
  11. Configure your ldap server as a replication provider with the syncrepl overlay: man slapo-syncprov.
    1. Make slapd load the 'syncprov' module. Stop the ldap server (it does not see to allow dynamic configuration of the list of modules), edit cn=config/cn=module{0}.ldif, add another olcModuleLoad attribute for syncprov and restart the server.
    2. Add a new olcSyncProvConfig entry for the 'syncprov' overlay, making it a child of the database entry 'olcDatabase={1}hdb,cn=config'. Set the olcSpReloadHint attribute to TRUE as suggested on the man page slapo-syncprov.

Markus' hints and tips

  • You will need these packages: slapd ldap-utils ldap-utils libpam-ldap libnss-ldap nscd.
  • Use ldapsearch to test your ldap server from the command line.

Advanced Challenge

  • Secure the connection to the LDAP server with TLS or SSL and a server certificate (we recommend 'tinyca2' (from 'tinyca' package) for certificate management).
  • Configure your L2 virtual machine as an LDAP replication slave, with your L1 acting as master. Establish a secure connection with ldaps:// or ldap:// + TLS.

Presentation

Mail, DNS

  • Date: 2012-05-22
  • Tutor: Laszlo Kajan
  • Topics: mail transport agents (MTA); procmail; Maildir and mbox formats; Internet message access protocol (IMAP); domain name server (DNS)

Questions

  • What is a mail transport agent (MTA)? - explain the role of MTAs on an example of sending an email.
  • What are the most popular MTAs (e.g. postfix, exim, sendmail)? Mention their strengths and weaknesses.
  • What is the internet message access protocol (IMAP) for?
  • What is procmail for? Show and explain examples of procmail 'recipes'.
  • What do domain name servers (DNS) do?
  • What is a DNS zone? How do you add zones to your DNS server's configuration?
  • Explain a simple DNS zone file. Show examples of forward (A) and reverse (PTR) address resolution resource records, show an example (and explain) a mail exchange (MX) record.

Slides

New topics: Ariane Bohm File:TblMailDNSArianeBoehm.pdf

Arianes Hints: nice Procmail Tutorial

Links for Preparation

Programming Challenge

  • Recommended to install: graphical desktop (gnome-core); iceweasel, icedove.
  • Set up, configure and use a bind9 DNS server.
  • Set up and configure a mail server (postfix recommended).
  • Set up an IMAP server (dovecot recommended).
  • Use Thunderbird / Icedove to send mail to another course member, configure the address book in Icedove/Thunderbird to connect to your LDAP server so that you can have your address book stored in LDAP.

Hints and Tips

  • packages to install: bind9, dnsutils; postfix, postfix-doc, bsd-mailx; dovecot-imapd; icedove; ca-certificates; procmail; tinyca2 (for advanced challenge);
  • Postfix configuration: choose 'Internet site'.
Name server
  • Edit /etc/bind/named.conf.local, add:
zone "tbl" {
       type master;
       file "/etc/bind/db.tbl";
};

zone "16.168.192.in-addr.arpa" {
       type master;
       file "/etc/bind/db.192.168.16";
};
  • Edit /etc/bind/db.tbl and /etc/bind/db.192.168.16, have:

/etc/bind/db.tbl:

;
; BIND data file for tbl zone
;
$TTL    86400
@       IN      SOA     lkajan.tbl. root.lkajan.tbl. (
                       12051501         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      lkajan.tbl.

lkajan          A       192.168.16.2
<other course members>

/etc/bind/db.192.168.16:

;
; BIND reverse data file for tbl zone
;
$TTL    86400
@       IN      SOA     lkajan.tbl. root.lkajan.tbl. (
                       12051201         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      lkajan.tbl.

2       PTR     lkajan.tbl.
<other course members>
  • Replace lkajan and 192.168.16.2 with your host name and IP in the SOA and NS records.
  • Test-load your named configuration: named-checkconf -z.
  • Restart the name server.
  • Update your /etc/resolv.conf with your own name server:
search tbl
nameserver 127.0.0.1
...
  • Test the name server with: host <name>.tbl; dig <name>.tbl; ping <name>.tbl.
Mail server
  • Basic configuration can be done simply with dpkg-reconfigure postfix. Config files of interest for manual configuration: /etc/postfix/main.cf, /etc/postfix/master.cf, /etc/aliases.
  • Edit your ~/.procmailrc and configure Maildir / mbox delivery as you prefer:
# Maildir:
DEFAULT="$HOME/Maildir/"
  • Send a mail to yourself as root.
  • Examine the mail log and check if the mail was delivered well.
Dovecot (IMAP server)
  • Edit /etc/dovecot/dovecot.conf: do not change anything but look at the protocols and the authentication: PAM does the work for us; ssl_cert_file and ssl_key_file: this is where you can secure communication to the server.
Thunderbird / Icedove
  • Start Icedove and configure a new mail server:
    • Email address: <username>@<hostname>.tbl.
    • Type: IMAP
    • Incoming server: <hostname>.tbl or 127.0.0.1
    • Outgoing server: <hostname>.tbl or 127.0.0.1
    • Configure LDAP: Preferences -> Composition -> Addressing -> Directory server -> Edit directories -> Add:
Hostname: localhost
Base DN: dc=tbl
Port n: 389
Bind DN: uid=<username>,ou=people,dc=tbl
    • Make sure your LDAP server serves connections to ldap://localhost/ (check in /etc/default/slapd).
    • Try sending a mail to another course member, e.g. Laszlo Kajan <lkajan@lkajan.tbl>.

Advanced challenge

  • Set up procmail recipes that automatically:
    • Reply to the sender that you are busy preparing to an exam if the mail subject contains the word 'work'.
    • Reply to the sender that you are busy with your work when the subject contains the work 'exam'.
    • Reply to the sender that you are ill when the subject contains both 'exam' and 'work'.
  • Create a postfix regular expression table for aliases and use this table to deliver all mail matching the pattern '/^sink/' to /dev/null.
  • Configure spamassassin for your MTA (postfix) or in .procmailrc.

Presentation

Programming challenge: Yannick Mahlich File:TBL-Mail DNS-programming challenge.pdf

Web server

  • Date: 2012-06-05
  • Tutor: Laszlo Kajan
  • Topics: Apache web server, common gateway interface (CGI), PHP

Questions

  • What is the HTTP protocol? Describe briefly (at least) the most common HTTP methods: GET, POST.
  • What does it mean that HTTP is a stateless protocol?
  • What are popular HTTP servers?
  • Mention ways to make web pages dynamic on the server side.
  • How is the common gateway interface (CGI) used to generate dynamic web pages?
  • How is PHP used to generate dynamic web pages?

Slides

New topics: Eva Reisinger File:Eva Reisinger webserver.pdf

Links for Preparation

Programming Challenge

  1. Install Apache.
  2. Create a simple web page with something like 'Hello world!' on it at http://yourname.tbl/ and make it reachable by other course members - well, you can test it from your L2 with say w3m (or lynx or wget or curl).
  3. Set up PHP for use with Apache and create a page that:
    • Takes a parameter 'name' and prints 'Hello <name>!'. You can use a form for setting the name if you want.
    • Calls the phpinfo() function.
    • Is available at http://yourname.tbl/test.php .
  4. Create a CGI program at http://yourname.tbl/test.cgi:
    • Take a parameter 'name' and print 'Hello <name>!'.
    • Print out the environment of the CGI program.
    • Print out the received parameters and their values.
  5. Install phpldapadmin and connect it to your LDAP server.
  6. Install ldap-account-manager and connect it to your LDAP server.
  7. Configure per-user web-accessible directories with mod_userdir.
  8. Set up HTTPS for the website:
    1. Create a server certificate with tinyca2 (preferably as root) from the tinyca package.
    2. Name your certificate authority (CA) "<uid>", e.g. "lkajan", when tinyca2 asks you.
    3. Copy your new CA certificate (/root/.TinyCA/<uid>/cacert.pem) into /usr/share/ca-certificates, name it <uid>.crt and make it readable by all. Execute dpkg-reconfigure ca-certificates and tick in your certificate to have it installed for your system. Also expose this CA certificate in web space at http://<uid>.tbl/<uid>.crt (a symlink in /var/www is enough).
    4. Set subject alternative names for the server certificate: 'IP:<your_IP_192.168.16.X>', 'DNS:<uid>.tbl' and 'DNS:www.<uid>.tbl' so that clients can recognize your server both by IP address and names. If you want to serve additional names or additional secondary IP addresses, also set these into the subject alternative name. In order to do this, use the menu 'Preferences/OpenSSL Configuration/Server Certificate Settings' and set 'Ask User' into the 'Subject alternative name (subjectAltName)' field (the top one), then click 'raw' below. Then, when you create the new server certificate (5th icon from the right, sorry, apparently no tooltips in this version :| ), set 'IP:<your_IP_192.168.16.X>,DNS:<uid>.tbl,...' into the Subject alternative name (at the request signing step). Do not add the eMail address to Subject DN.
    5. Export your web server key and certificate in one file in PEM format without passphrase into a file in /etc/apache2. You will have to be on the 'Keys' tab to do this. Make sure this file is only readable by root! Set this file into 'SSLCertificateFile' in the configuration of the secure site (default-ssl). Comment 'SSLCertificateKeyFile' out.
    6. Enable Apache site 'default-ssl' and module 'ssl'.
    7. Point your browser at http://<uid>.tbl/<uid>.crt , import and trust your CA certificate.
    8. Make sure the php and CGI pages open properly both with http:// and with https:// protocols.
    9. Observe how the https:// connection to your site is now trusted and verified by your own certificate. Point your browser at http://<uid>.tbl/<uid>.crt of other course members, install and trust their certificate and then visit their sites securely (with https). Your browser should not complain about untrusted connections after you install their CA certificates. Also visit their sites by IP address. Your browser should accept this as secure as well, due to the IP address in the subjectaltname of the certificate.
  9. Create a secure section of the web site that requires authentication: https://<uid>.tbl/secure/index.html . Make this area accessible only via HTTPS. Authenticate against your LDAP database. Implement this in /var/www/secure/.htaccess.
    1. You will have to set the appropriate AllowOverride level for this to work.
    2. You have to enable (a2enmod) the authnz_ldap module for ldap authentication.

Hints and Tips

  • /var/log/apache2/error.log is your friend. Also use the Apache documentation. I find this page very useful.
  • The package for PHP is called libapache2-mod-php5. Installing this will pull in apache2 as well.
  • You can use bluefish to create and edit a web page in a GUI.
  • You can use the Perl CGI module for your CGI program. Alternatively you can use C/C++ to solve this challenge for fun... hmm.
  • One way to make the CGI script accessible at http://lkajan.tbl/test.cgi is to allow the execution of CGI scripts in the document root (with the ExecCGI option) and add/set the 'cgi-script' handler[6] for files with '.cgi' extension.
  • Start tinyca2 as root. When it is run for the first time, it asks you to fill in data to create a new certificate authority (CA). You will use this CA to issue a certificate for your web server. Fill in the fields you understand (e.g.. State: "Bayern"), examine the other fields.

Presentation

Programming challenge: Simon Domke File:Presentation webservers.pdf

Databases and SQL

  • Date: 2012-06-12
  • Tutor: Laszlo Kajan, Christian Schaefer
  • Topics: MySQL daemon setup, tweaking the server, user management

Questions

  • What are the differences between Excel and a DBMS?
  • What are famous DBMSs?
  • What advantages does data storage in a DB have over simply putting data into a flat file?
  • What ways exist to access a MySQL database?
  • What ways exist to backup a MySQL database?

Slides

New topics: Benjamin Drexler File:Biolab ss12 dbms drexler.pdf

Links for Preparation

Programming Challenge

  1. Install and configure a MySQL Database server
  2. Make yourself familiar with basic user management
    1. Create a user u1 and a database db1
      1. Grant u1 full access to db1
    2. Create a second user u2 and give her only read access
    3. How can you retrieve all rights of a given user?
    4. How can you take privilidges away from a user?
  3. Create a table in your database db1 with three columns
    1. The table should contain a primary key that spans two columns
    2. For example use PHP or Perl or Python.
  4. Create a backup from your database.
  5. Read-out the basic server status
    1. How many concurrent client connections are maximally allowed?
    2. How can you find out about the actual established connections?
  6. Write a script that establishes n concurrent connections and monitor the server status.
    1. Raise the maximally allowed connections to the server when you reach the limit.

Hints and Tips

  • Install package: mysql-server
  • Which additional packages will be installed?
  • Which client programs?
  • Familiarize yourself with the default configuration file /etc/my.cnf
  • What is the TCP/IP Port the MySQL server/ client applications will listen to?
  • How many concurrent sessions the MySQL server will allow?
  • What is the size of the query cache used to cache SELECT results?

Advanced Challenge

  1. Fill your table with several million entries (mind the primary key!)
  2. Write a script that establishes several concurrent connections.
    1. During each connection, the script should conduct several advanced SELECT-queries.
    2. Monitor the CPU- and memory usage of your MySQL server during script-lifetime.
    3. Could you think of server variables whose tweaking lead to a performance gain?
  3. Backup the whole database using
    1. mysqldump and
    2. the mysql command
  4. What are storage engines? Which are there and what are their differences?

Presentation

Programming challenge: Christian Mertes File:Christian Mertes mysql exercise.pdf

Web Content Management Systems

  • Date: 2012-06-19
  • Tutor: Laszlo Kajan
  • Topics: web content management systems: MediaWiki, Drupal; bug tracking/software development management.

Questions

  • What is a web content management system?
  • How does a web content management system help you maintain a website?
  • Highlight the differences between Drupal and MediaWiki: what for and when would you use which?
  • What is Bugzilla? Can Bugzilla be used as a general request tracker?

Slides

New topics: Yannick Mahlich File:TBL CMS Talk.pdf

Links for Preparation

Programming Challenge

  • Install MediaWiki.
  • Install the CMS of your choice (Drupal recommended).
  • Install Bugzilla.
  • Connect the user management of the CMS, wiki and Bugzilla to your LDAP.
  • Create a simple web page with your CMS for the practical.
  • Create a wiki page that is editable after login by users of your machine, but not by the world.
  • Create products and components in Bugzilla for your wiki and CMS. Allow everybody to file bugs.

Hints and Tips

Wiki

There are many different wiki engines:

We are going to use MediaWiki (http://www.mediawiki.org), one of the most popular wiki engines available.

1. Install a good and stable debian package: mediawiki

2. Adjust the MediaWiki configuration file to the system environment

  • add to your virtual host file
Include /etc/mediawiki/apache.conf 
  • do not forget to reload the Apache
  • which domain do you now use to access the mediawiki?
  • Uncomment the third line in /etc/mediawiki/apache.conf, so that line reads
Alias /mediawiki /var/lib/mediawiki
  • the alias can be replaced with any other alias you want

3. Complete the installation settings over the Internet

  • discuss with your neighbor a suitable configuration

4. Review the settings in the default and the local configuration files

  • the default configuration file should not be edited
  • what permissions do you set for the 'LocalSettings.php' file?'

5. Modify the main page to make it a little more personal and at least add a logo. Allow registered users to change the content ($wgGroupPermissions).

  • Enable LDAP Authentication for your MediaWiki (debian package: mediawiki-extensions-ldapauth)

Advanced Challenge

Presentation

Programming challenge: Benjamin Drexler File:Biolab12 cms challenge drexler.pdf

Network Filesystems and Grid Computing

  • Date: 2012-06-26
  • Tutor: Laszlo Kajan
  • Topics: filesystem sharing (NFS, SMB/CIFS and sshfs), batch-queueing/high throughput computing (Grid Engine)

Questions

  • List use cases for file system sharing - what is this good for?
  • How would you make the same file system available to Linux, Windows and OS X clients?
    • Can you make your home file system available to a Windows virtual machine guest running on your Linux host?
  • How would you make a file system available to Linux clients where high performance is important? (hint: NFS, but also think of OCFS2 [7])
  • Can you browse a remote file system as if it were local in case it is on a host with SSH access?
  • Is it possible to combine multiple hosts to serve out a single file system (c.f. OCFS2, GlusterFS)?
  • What are the most popular batch-queueing/grid computing engines?
  • What is a batch system like the Open Grid Scheduler good for?
    • What happens when more jobs are submitted than the number of available cores?
    • Is it possible to prioritize users?
  • Can the Open Grid Scheduler/Grid Engine handle non-uniform (say some have 32G memory, others 64G) execution hosts?
    • Is it possible to match execution hosts to job requirements (e.g. memory, number of cores, installed software)? Show examples.
  • When the (Sun) Grid Engine is installed on a cluster, it is common that all execution hosts mount at least one shared file system, say /mnt/home. Why do you think this is?

Slides

New topics: Christian Mertes File:Grid computing.pdf

Links for Preparation

Cluster Filesystems

Object Store

Programming Challenge

  • Install the Linux kernel NFS server (nfs-kernel-server package, make sure you have the nfs-common package installed as well).
    • Export (man exports) - with NFS (up to version 4) - the root of your home directory (/home) read-only, 'root_squash' and 'all_squash' to the world ('*').
    • Export the root of your home directory (/home) read-write, no squash (not even root) to your L1 and L2 virtual machine, as well as to 127.0.0.1 (localhost).
    • Mount your exported home directory to /mnt/<uid>-home/, either in your L1 or L2: man mount. Use the 'nfs4' protocol. Enter the mounted file system into /etc/fstam, but with the 'noauto' option.
  • Install the SMB/CIFS server (samba package). I recommend you also install the SMB client (smbclient package) and the samba-doc package. Workgroup/Domain name: 'TBL'.
    • Unfortunately the UNIX passwords are not usable for Samba. UNIX and Samba encrypted passwords have to be kept separately. Use 'smbpasswd -a <user>' to set the Samba password for your user - but look out: this sets your UNIX password as well (with the default smb.conf file).
    • Create a new share 'clipboard' (man smb.conf) that exports /srv/samba/clipboard read-write to valid users. Make sure /srv/samba/clipboard is writable to everyone. Make the share browseable and force files and directories created to be modifiable by all (force create and directory mode 0777).
    • Start a file manager in the GUI (X) of your L1 virtual machine and browse the windows network: find your clipboard share and log in to it. Create/copy a file in a new directory.
  • Install the SSH filesystem client (sshfs package) in your L1 virtual machine. Put yourself into the 'fuse' group.
    • Use sshfs (man sshfs) to mount your home directory on i12r-tbl (tbl:/home/tbl2012/<yourname>) to ~/L0home on the L1. You will probably have to use id mapping: '-o idmap=user'. Create/copy a file in a new directory in ~/L0home.
    • Use 'fusermount -u <mount_point>' to unmount the sshfs file system.
  • Install the (Sun) Grid Engine (packages gridengine-{client,exec,master,qmon}, you will also need xfonts-100dpi and xfonts-75dpi (thanks Daniel), but you probably have these already). Let debconf configure SGE automatically. SGE cell name: <uid> (e.g. lkajan) from the table above. Mater host: <host name> (e.g. lkajan.tbl) from the table above.
    1. Start (as root) the qmon graphical management interface (forward X or in a VNC session).
    2. Add '<yourhost.tbl>' as a submit host (under Host Configuration button).
    3. Create a new queue (from Queue Control) 'default' with shell '/bin/sh' instead of '/bin/csh'. Add '<yourhost>.tbl' to its Hostlist.
    4. Add yourself to the list of users (User Configuration, User tab). Set the total number of Share Tree tickets (Policy Configuration) to 10,000 (10k) and give yourself 1000 tickets in the Share Tree Policy (Share Tree Policy -> Add Leaf (to Root node - add root first)). Configure the total number of tickets to be distributed among unspecified users to 1000 (Add Leaf to root, give Name = 'default').
    5. Submit the binary job /bin/date with 'qsub' (man qsub). Join the standard output and error of the job. Where do you get the output of this job?
    6. Write a simple job script that calls /bin/date, but have this job script define the necessary arguments for joining standard out and standard error, so that there is no need to give these on the command line (hint: man qsub, search for '#$'). Submit this job script.

Hints and Tips

  • Use a bind mount (man mount) to make /home available in /srv/nfs4/home.
  • Use 'exportfs -v' to check what, and how is exported.
  • Restart service nfs-common after making changes to /etc/exports or /etc/fstab. Setting 'NEED_IDMAPD=yes' in '/etc/default/nfs-common' may also be helpful to avoid confusion. If you do not get proper user/group mapping for your NFSv4 mount, the lack of a running rpc.idmapd daemon may be the cause.
  • Use 'smbclient -U <user> -L //<yourhost>.tbl/' to check what is exported by the Samba server for a particular user.

Advanced Challenge

  • Configure the ldapsam authentication backend for the Samba server.
  • Mount (-t cifs) your Samba clipboard share to your L2 virtual machine (cifs-utils package, man mount.cifs).
  • Turn your L2 virtual machine into an SGE execution host. Share your home file system with the L2 so that your jobs can access this file system easily.

Presentation

Programming challenge: Daniel Bader File:Daniel tbl2012 nfs-grid-challenge.pdf

Packaging for major Linux distributions and Debianization of PredictProtein

This session introduces the topics and assignments for the final weeks of the course. The students are to work on their assignments intensively under the supervision of the tutor during the remaining practical sessions. Full time attendance of these sessions is a must.

  • Date: 2012-07-03 (and 2012-07-10)
  • Tutor: Laszlo Kajan
  • Topics: RPM and Debian packages overview, Debian Social Contract, Debian Policy, Debian Med Pure Blend, creating Debian packages, Debian package quality control, contributing packages to Debian

Questions

  • What is the advantage of an RPM package over a tar.gz package? What is the advantage of a Debian (deb) package over an RPM package (if any)?
  • What type of package do popular commercial (enterprise) Linux distributions use (e.g. RedHat Enterprise Linux, SUSE Linux Enterprise)?
  • What is the Debian Social Contract and the Debian Free Software Guidelines?
  • What is the Debian Policy?
  • What are the archive areas in Debian?
  • What are the major steps of 'Debianizing' a piece of software?
  • What is the tool 'debhelper' good for?
  • What is the role of the tool 'lintian' in packaging for Debian?
  • What is the role of the 'quilt' tool in packaging for Debian?
  • Who are ftpmasters?
  • What is PredictProtein (The PredictProtein server, NAR, 2003)?

Slides

New topics: Cedric Staniewski File:Debian packaging Cedric.pdf and Christof Angermueller File:Dpkg angermueller.pdf

Links for Preparation

Programming Challenge

Package Upstream Uploader Comment
predictnls ftp://rostlab.org/predictnls/ Julia Accepted
librg-exception-perl ftp://rostlab.org/librg-exception-perl/ Ariane Accepted
libnhgri-blastall-perl ftp://ftp.nhgri.nih.gov/pub/software/blastall/ NHGRI-Blastall-[0-9.]+.tar.gz Jens Accepted
profbval ftp://rostlab.org/profbval/ Christof Accepted
profisis ftp://rostlab.org/free/ profisis-[0-9.]+.tar.gz Cedric Accepted
norsnet ftp://rostlab.org/norsnet/ Christian Accepted
libai-fann-perl http://search.cpan.org/dist/AI-FANN/ Yannick Already in Debian. Moved from Med to Perl Team.
librg-liu-bundle-perl ftp://rostlab.org/free/ librg-liu-bundle-perl-[0-9.]+.tar.gz Daniel not a dep ITP 680960 closed
librg-pp-bundle-perl ftp://rostlab.org/free/ librg-pp-bundle-perl-[0-9.]+.tar.gz Benjamin not a dep ITP 680996 closed
norsp ftp://rostlab.org/norsp/ Eva Accepted
predictprotein ftp://rostlab.org/predictprotein/ Simon Accepted

You will be randomly assigned already Debianized PredictProtein packages. Transfer the Debianization to Alioth, review the Debianization, fix all issues and commit your version. Address the tutor's and other Debian Med members' comments as appropriate. Get your packages ready by the 23rd of July.

  1. You should sign up to debian-med-packaging@lists.alioth.debian.org while you work on the packages. You can also sign up to debian-med@lists.debian.org.
  2. Read and follow the Policy of the Debian Med Team.
  3. File an intention to package (ITP) bug against the 'wnpp' (work-needing and prospective packages) pseudo package for your package on L0 with reportbug --email <your@email> wnpp. Details of your package for the bug report can be found in the upstream package and in the Debianization you received.
  4. Debianization should be done on the unstable distribution. Setting this distribution up on your virtual machine (that runs stable) is going to be a lot harder than the packaging itself. If it is too hard, I could create an account for you on a server running Debian unstable. Just let me know.
    1. Install Debian unstable on your L1 into a chroot with debootstrap unstable /srv/unstable. Execute the command chroot /srv/unstable (as root) to get into this 'chroot environment'. Exiting the shell takes you back to your original location.
    2. In order to make your packaging life easier, bind-mount /home into /home of the chroot, mount /proc as well, e.g. add this to /etc/fstab (on your L1 outside the chroot environment):
      proc /srv/unstable/proc proc defaults 0 0
      /home /srv/unstable/home none bind 0 0
      Execute mount -a.
    3. You have to make your chroot environment know about you as well. Either create your user in that environment, or configure connection to your LDAP server: make sure your (L1, non-chroot) LDAP server serves also via TCP (ldap:///), then go through the steps you did before to configure the LDAP client, but do not install nscd in the chroot environment (it confuses things). Connect libnss-ldap and libpam-ldap to ldap://127.0.0.1/.
  5. chroot into the 'unstable' chroot environment and become yourself: chroot /srv/unstable/ su - lkajan.
  6. cd into the directory where your packaging work will take place.
  7. For the following to work, the SSH key (not GPG, not X.509 key; generate with ssh-keygen) you have on your L1 will have to be uploaded to Alioth, into the text area on My Page/Account Maintenance/Shell Account Information/Edit Keys. It may take an hour for your key to become usable! Alternatively you can copy an already uploaded SSH key pair into your L1.
  8. Use svn (or git, but from here on I assume you use svn) to create a directory for your work in the versioning system on Alioth:
    svn mkdir --parents svn+ssh://<you-guest>@svn.debian.org/svn/debian-med/trunk/packages/<yourpackage>/trunk svn+ssh://<you-guest>@svn.debian.org/svn/debian-med/trunk/packages/<yourpackage>/tags -m 'repository directory for <yourpackage> -- <a short description of it, copy from ITP bug>'
    Do not create the 'branches' directory in the repository: it is not necessary.
  9. Check out svn+ssh://<you-guest>@svn.debian.org/svn/debian-med/trunk/packages/<yourpackage> and cd into trunk. Untar the debian directory from the Debianization tarball I sent you here. You should have only one directory 'debian' in 'trunk'.
  10. While in trunk, mkdir ../tarballs ../build-area. Do not check these directories into the versioning system (use svn:ignore property to hide them from an svn status).
  11. We are not going to put the entire upstream into the repository. We are going to do merging of the upstream and the Debianization at build time. Set the special flag mergeWithUpstream to 1 on 'debian':
    svn pset mergeWithUpstream 1 debian
  12. Now the real packaging work begins.
    1. Make sure you have dpkg-dev and packaging-dev installed.
    2. Review all files in 'debian' and add missing ones. Use the Policy as your ultimate reference. You may find the Debian New Maintainers' Guide useful as well.
  13. You do not yet have the upstream package in its proper place. Perhaps it is best to write debian/watch first and use uscan --verbose --destdir=../tarballs --force-download to get the upstream tarball. This way you make sure the watch file works.
  14. Things to look out for in debian directory:
    1. Create debian/source/format if missing, with 3.0 (quilt) in it.
    2. Set debian/compat to 8 in order to allow building on 'stable'.
    3. debian/copyright has to conform to [8]. Use cme fix dpkg-copyright (from libconfig-model-perl) to check and config-edit -application dpkg-copyright debian/copyright to edit - if vim is not enough.
    4. Fill in the debian/upstream file according to [9]. Multiple references are (or will be) supported. Examples: [10] [11]. You can use this link to validate the YAML in debian/upstream.
  15. When you have the upstream downloaded and you are ready to build the package:
    1. svn-buildpackage --svn-reuse --svn-ignore builds the package in ../build-dir. Do not worry about not being able to sign the package (ignore this error or use '-us -uc'). Have 'svn-lintian' in ~/.svn-buildpackage.conf in order to run lintian automatically.
    2. If you want to test the build and apply patches, cd ../build-area/<package-build-dir>/. There remove(!) the debian directory and replace it with a symlink to ../../trunk/debian (ln -s ../../trunk/debian). Use debuild -b -us -uc to test the building.
    3. If you use quilt (for patching), have this in your ~/.quiltrc:
      QUILT_PATCH_OPTS="--reject-format=unified"
      QUILT_DIFF_ARGS="-p ab --no-timestamps --no-index --color=auto"
      QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
      Quilt patches go to debian/patches. You will probably want to export QUILT_PATCHES=debian/patches.
  16. The package(s) you get should come with no lintian errors in them. Fix all lintian warnings reported with the flags --display-experimental --display-info --pedantic. Configure lintian like this:
    ~/.lintianrc:
    color=always
    display-experimental=yes
    display-info=yes
    pedantic=yes
    show-overrides=yes
  17. Remember to close the ITP bug for your package in debian/changelog, like (Closes: #<ITPBUG>).

Hints and Tips

  • Packages you need: dpkg-dev, packaging-dev

Advanced Challenge

  • Prepare the RPM package of one of your assigned packages. A CentOS virtual machine is useful here.

Presentation

Programming challenge: Cedric Staniewski File:Packaging profisis.pdf and Jens Preussner File:Debian-Perl.pdf

Packaging for major Linux distributions and Debianization of PredictProtein (contd.)

  • Date: 2012-07-10
  • Tutor: Laszlo Kajan
  • Topics: Preparation of PredictProtein packages continued.

Computer clusters and external Services: excursion to the LRZ

  • Date: 2012-07-17 16:00-17:00
  • Please bring a photo ID with you. Also, it may be cold in the computer rooms, be prepared.

Evaluation and Upload of Debian Packages

This is the deadline for preparing your Debian packages. Commit your best by this date.

  • Date: 2012-07-23
  • Tutor: Laszlo Kajan
Personal tools