Difference between revisions of "Phylogenetics: Xanadu Cluster"

From EEBedia
Jump to: navigation, search
(Man pages)
 
(17 intermediate revisions by the same user not shown)
Line 13: Line 13:
 
You will be using a couple of simple (and free) programs to communicate with the head node of the cluster.  
 
You will be using a couple of simple (and free) programs to communicate with the head node of the cluster.  
  
 +
=== If you use Windows...please scroll down to the Windows section===
 
=== If you use MacOS 10.x... ===
 
=== If you use MacOS 10.x... ===
 
==== SSH ====
 
==== SSH ====
Line 30: Line 31:
 
in your terminal (the initial cd command changes the directory to your default, a.k.a. home, directory).
 
in your terminal (the initial cd command changes the directory to your default, a.k.a. home, directory).
  
If you do not see a file named "config" in your ".ssh" director, create an empty config file using the command  
+
If you do not see a file named "config" in your ".ssh" director, create an empty config file using the command  
 
   touch config
 
   touch config
Open the config file in a text editor such as [https://www.barebones.com BBEdit] (NOT a word processor such as Microsoft Word or Pages!) and add the following lines (replace ''username'' with your actual username on xanadu, and replace ''homedir'' with your home directory path on your Mac - for example, my homedir is "/Users/plewis"):
+
Open the config file in a text editor such as [https://www.barebones.com BBEdit] (NOT a word processor such as Microsoft Word or Pages!) and add the following lines (replace ''plewis'' with your actual username on xanadu, and replace ''/Users/plewis'' with your home directory path on your Mac):
 
  host xanadu
 
  host xanadu
 
     HostName xanadu-submit-ext.cam.uchc.edu
 
     HostName xanadu-submit-ext.cam.uchc.edu
     User username
+
     User plewis
     IdentityFile homedir/.ssh/id_rsa
+
     IdentityFile /Users/plewis/.ssh/id_rsa
 +
    Cipher blowfish
 +
host xfer
 +
    HostName transfer.cam.uchc.edu
 +
    User plewis
 +
    IdentityFile /Users/plewis/.ssh/id_rsa
 
     Cipher blowfish
 
     Cipher blowfish
 
Once you save the config file, you should be able to just type
 
Once you save the config file, you should be able to just type
 
   ssh xanadu
 
   ssh xanadu
to login to the xanadu cluster.
+
to login to the xanadu cluster. (The second entry (xfer) will be used when transferring files to and from the cluster.)
  
 
==== SCP/SFTP ====
 
==== SCP/SFTP ====
Line 57: Line 63:
 
'''Creating a shortcut'''
 
'''Creating a shortcut'''
  
If you want to avoid having to type the long command above every time you want to connect to the cluster, it is possible to create a shortcut. You will need to edit the ''config'' file in your ''.ssh'' directory. Open (or create, if it does not yet exist) the file named ''config'' in a text editor such as [https://notepad-plus-plus.org NotePad++] (NOT a word processor such as Microsoft Word!) and add the following lines (replace ''username'' with your actual username on xanadu, and replace ''homedir'' with your home directory path on your Mac - for example, my homedir is "/Users/plewis"):
+
If you want to avoid having to type the long command above every time you want to connect to the cluster, it is possible to create a shortcut. You will need to edit the ''config'' file in your ''.ssh'' directory. Open (or create, if it does not yet exist) the file named ''config'' in a text editor such as [https://notepad-plus-plus.org NotePad++] (NOT a word processor such as Microsoft Word!) and add the following lines (replace ''plewis'' with your actual username on xanadu, and replace ''/c/Users/Paul Lewis'' with your home directory path):
 
  host xanadu
 
  host xanadu
 
     HostName xanadu-submit-ext.cam.uchc.edu
 
     HostName xanadu-submit-ext.cam.uchc.edu
     User username
+
     User plewis
     IdentityFile homedir/.ssh/id_rsa
+
     IdentityFile "/c/Users/Paul Lewis/.ssh/id_rsa"
 
     Cipher blowfish
 
     Cipher blowfish
 +
host xfer
 +
    HostName transfer.cam.uchc.edu
 +
    User plewis
 +
    IdentityFile "/c/Users/Paul Lewis/.ssh/id_rsa"
 +
    Cipher blowfish
 +
Use the <tt>pwd</tt> command to find out what your home directory path is, and use double quotes if your home directory path contains embedded spaces (note that I had to use quotes for mine).
 +
 
Once you save the config file, you should be able to just type
 
Once you save the config file, you should be able to just type
 
   ssh xanadu
 
   ssh xanadu
to login to the xanadu cluster.
+
to login to the xanadu cluster. (The second entry (xfer) will be used for transferring files to and from Xanadu.)
 
+
= THIS PAGE IS NOT ACCURATE BELOW THIS POINT - IT IS IN THE PROCESS OF BEING REWRITTEN =
+
  
 
==== SCP/SFTP ====
 
==== SCP/SFTP ====
Line 128: Line 139:
  
 
==== Create the gopaup file ====
 
==== Create the gopaup file ====
Now use nano to create a second file named <tt>gopaup</tt> in your home directory (the parent directory of the <tt>pauprun</tt> directory). To do this, type <tt>nano gopaup</tt>. This file should contain this text:
+
Now use nano to create a second file named <tt>gopaup</tt> in your <tt>pauprun</tt> directory. To do this, type <tt>pwd</tt> to make sure you are in the <tt>pauprun</tt> directory, then type <tt>nano gopaup</tt>. This file should contain this text:
#$ -S /bin/bash
+
  #!/bin/bash
#$ -o junk.txt -j y
+
  #SBATCH --partition=general
cd $HOME/pauprun
+
  #SBATCH --qos=general
module load paup/current
+
  #SBATCH --job-name=pauprun
paup -n run.nex
+
  cd $HOME/pauprun
 +
  module load paup/4.0a-166
 +
  paup -n run.nex
  
 
=== Using Cyberduck to upload the algae.nex data file ===
 
=== Using Cyberduck to upload the algae.nex data file ===
[[File:Cyberduck bookmark.png|right]]  
+
[[File:Cyberduck-bookmark-xanadu.png|right]]  
 
Download the file <tt>algae.nex</tt> from [http://hydrodictyon.eeb.uconn.edu/people/plewis/downloads/algae.nex here] and save it on your hard drive.
 
Download the file <tt>algae.nex</tt> from [http://hydrodictyon.eeb.uconn.edu/people/plewis/downloads/algae.nex here] and save it on your hard drive.
  
Open Cyberduck, choose Bookmark > New Bookmark from the main menu, then fill out the resulting dialog box as shown on the right (except substitute your own user name, of course). Be sure to change the protocol to SFTP (not the default FTP). Click the X button to close and you should see your bookmark appear at the bottom of the main window. Double click the bookmark to open a connection. You will then be warned that the host key is unknown - choose Allow (and go ahead and check the Always button so you do not need to do this every time.
+
Open Cyberduck, choose Bookmark > New Bookmark from the main menu, then fill out the resulting dialog box as shown on the right (except substitute your own user name, of course). Be sure to change the protocol to SFTP (not the default FTP). Click the button to close the dialog box and you should see your bookmark appear at the bottom of the main window. Double click the bookmark to open a connection. You will then be warned that the host key is unknown - choose Allow (and go ahead and check the Always button so you do not need to do this every time.
  
 
Once you are in, you will see a listing of the files in your home directory (if you have any). To copy the <tt>algae.nex</tt> file to the cluster, you need only drag-and-drop it onto the Cyberduck window.
 
Once you are in, you will see a listing of the files in your home directory (if you have any). To copy the <tt>algae.nex</tt> file to the cluster, you need only drag-and-drop it onto the Cyberduck window.
  
===  (Mac/Linux users only) Using scp to upload the algae.nex data file ===
+
===  Using scp to upload the algae.nex data file ===
 +
 
 +
While you will probably want to do your file transfers with Cyberduck as described above, it is also possible to transfer files using the command line scp client. Read on if you are interested in this option, but feel free to skip this section if you are happy using Cyberduck.
  
Mac users have two options. While you will probably want to do your file transfers with Cyberduck as described above, it is also possible to transfer files using the command line scp client (scp is also a good option for Linux users). Read on if you are interested in this option, but feel free to skip this section if you are happy using Cyberduck.
+
In your terminal, navigate to where you saved the file. If you saved it on the desktop, you can go there by typing <tt>cd Desktop</tt>.
  
Open the Terminal  application and navigate to where you saved the file. If you saved it on the desktop, you can go there by typing <tt>cd Desktop</tt>.
+
If you've made a shortcut in your ''.ssh/config'' file, you can use the following command to upload the ''algae.nex'' file:
 +
  scp algae.nex xfer:
  
Type the following to upload algae.nex to the cluster:
+
If you have not made a shortcut, use this command instead:
  scp algae.nex username@bbcsrv3.biotech.uconn.edu:
+
  scp algae.nex username@transfer.cam.uchc.edu:
where <tt>username</tt> should be replaced by your own user name on the cluster. (Don't forget the colon on the very end of the line!)
+
where <tt>username</tt> should be replaced by your own user name on the cluster. (Don't overlook the colon on the very end of the line!)
  
 
=== A few more UNIX commands ===
 
=== A few more UNIX commands ===
  
You have now transfered a large file (algae.nex) to the cluster, but it is not in the right place. The algae.nex file is currently in your home directory, whereas the run.nex file is in the pauprun directory. The run.nex file contains a line containing the command <tt>execute algae.nex</tt>, which means that algae.nex should also be located in the pauprun directory. Use the following commands to ensure that (1) you are in your home directory, and (2) algae.nex is also in your home directory:
+
You have now transferred a data file (algae.nex) to the cluster, but it is not in the right place. The algae.nex file is currently in your home directory, whereas the run.nex file is in the pauprun directory. The run.nex file contains a line containing the command <tt>execute algae.nex</tt>, which means that algae.nex should also be located in the pauprun directory. Use the following commands to ensure that (1) you are in your home directory, and (2) algae.nex is also in your home directory:
 
   cd $HOME
 
   cd $HOME
 
   ls algae.*
 
   ls algae.*
Line 174: Line 190:
 
  cd
 
  cd
 
  rm algae.nex
 
  rm algae.nex
The first cd command just ensures that the copy you are removing will be the one in your home directory (typing <tt>cd</tt> by itself acts the same as typing <tt>cd $HOME</tt>). If it bothers you that the system always asks your permission before deleting a file, you can force the issue using the -f option (but just keep in mind that this is more dangerous):
+
The first cd command just ensures that the copy you are removing will be the one in your home directory (typing <tt>cd</tt> by itself acts the same as typing <tt>cd $HOME</tt>).  
rm -f algae.nex
+
To delete an entire directory (don't try this now!), you can add the -rf flags. The r flag tells rm to recursively apply the remove command to everything in every subdirectory, while the f flag means force (don't ask whether each file should be deleted in each subdirectory, just do it!):
To delete an entire directory (don't try this now!), you can add the -r flag, which means to recursively apply the remove command to everything in every subdirectory:
+
 
  rm -rf pauprun
 
  rm -rf pauprun
The above command would remove everything in the pauprun directory (without asking!), and then remove the pauprun directory itself. I want to stress that this is a particularly dangerous command, so make sure you are not weary or distracted when you use it! Unlike the Windows or Mac graphical user interface, files deleted using rm are not moved first to the Recycle Bin or Trash, they are just gone. There is no undo for the rm command.
+
The above command would remove everything in the pauprun directory (without asking!), and then remove the pauprun directory itself. I want to stress that this is a particularly dangerous command, so make sure you are not distracted or sleep-deprived when you use it! Unlike the Windows or Mac graphical user interface, files deleted using rm are '''not''' moved to the Recycle Bin or Trash, they are just gone. There is '''no undo''' for the rm command.
  
 
=== Starting a PAUP* analysis ===
 
=== Starting a PAUP* analysis ===
  
 
If you've been following the directions in sequence, you now have two files (algae.nex and run.nex) in your <tt>$HOME/pauprun</tt> directory on the cluster, whereas the gopaup file should be in <tt>$HOME</tt>. Use the cd command to make sure you are in your home directory, then the cat command to look at the contents of the gopaup file you created earlier. You should see this:  
 
If you've been following the directions in sequence, you now have two files (algae.nex and run.nex) in your <tt>$HOME/pauprun</tt> directory on the cluster, whereas the gopaup file should be in <tt>$HOME</tt>. Use the cd command to make sure you are in your home directory, then the cat command to look at the contents of the gopaup file you created earlier. You should see this:  
  #$ -S /bin/bash
+
  #!/bin/bash
  #$ -o junk.txt -j y
+
  #SBATCH --partition=general
 +
#SBATCH --qos=general
 +
#SBATCH --job-name=pauprun
 
  cd $HOME/pauprun
 
  cd $HOME/pauprun
  module load paup/current
+
  module load paup/4.0a-166
 
  paup -n run.nex
 
  paup -n run.nex
This file will be used by software called the Sun Grid Engine (SGE for short) to start your run. SGE provides a command called qsub that you will use to submit your analysis. The SGE qsub command will look for a core (i.e. processor) on a node (i.e. machine) in the cluster that is currently not being used and will start your analysis on that node. This saves you the effort of looking amongst all nodes in the cluster for a core that is not busy.
+
 
 +
This file will be used by software called SLURM to start your run. SLURM provides a command called <tt>sbatch</tt> that you will use to submit your analysis. The SLURM <tt>sbatch</tt> command will look for a core (i.e. processor) on a node (i.e. machine) in the cluster that is currently not being used and will start your analysis on that node. This saves you the effort of looking amongst all nodes in the cluster for a core that is not busy.
  
 
Here is an explanation of each of the lines in gopaup:
 
Here is an explanation of each of the lines in gopaup:
* Lines beginning with the two characters #$ are interpreted as commands by SGE itself. In this case, the first #$ command tells SGE to interpret what follows as a bash script and the second #$ command causes SGE to send any output from the program to a file named junk.txt, and the -j y part says to append any error output to this as well (the j stands for join and the y for yes)
+
* The 1st line specifies the command interpreter to use (just include this in your scripts verbatim).
* The third line is simply a cd command that changes the present working directory to the pauprun directory you created earlier. This will ensure that anything saved by PAUP* ends up in this directory rather than in your home directory. Note that $HOME is like a macro that will be expanded to the full path to your home directory.
+
* The 2nd, 3rd, and 4th lines begin with #SBATCH and are interpreted as commands by SLURM itself. In this case, the first and second #SBATCH commands tell SLURM to use the general partition (--partition=general) and the general quality of service (--qos=general). You should always include these two lines verbatim. The last #SBATCH line gives a name to your job (--job-name=pauprun). You could change "pauprun" here to something else, but keep your job names short and without embedded spaces or punctuation. The job name will help you identify your run when checking status.
* The fourth line sets environmental variables such that invoking paup starts the most recent installed version of the program. If you left this line out you would end up running an older version of paup.
+
* The 5th line is simply a cd command that changes the present working directory to the pauprun directory you created earlier. This will ensure that anything saved by PAUP* ends up in this directory rather than in your home directory. Note that $HOME is like a macro that will be expanded to the full path to your home directory.
* The fifth and last line starts up PAUP* and executes the run.nex file. The -n flag tells PAUP* that no human is going to be listening or answering questions, so it should just use default answers to any questions it needs to ask during the run.
+
* The 6th line informs the system that you want to use a particular version of paup. If you left this line out, the command on the last line might not work at all, or might run an older version of paup. You can get a list of all available modules using the command "module avail"
 +
* The 7th and last line starts up PAUP* and executes the run.nex file. The -n flag tells PAUP* that no human is going to be listening or answering questions, so it should just use default answers to any questions it needs to ask during the run.
  
==== Submitting a job using qsub ====
+
==== Submitting a job using sbatch ====
Now you are ready to start the analysis. Make sure you are in your home directory, then type
+
Now you are ready to start the analysis. Type these commands to start your run:
  qsub gopaup
+
cd
 +
cd pauprun
 +
  sbatch gopaup
  
==== Checking status using qstat ====
+
==== Checking status using squeue ====
You can see if your run is still going using the qstat command:
+
You can see if your run is still going using the squeue command:
  qstat
+
  squeue
If it is running, you will see an entry containing gopaup and the state will be r (running), or qw (queued, waiting). Here is what it looked like for me:
+
If it is running, you will see an entry named pauprun. Here is what it looked like for me:
  [plewis@bbcsrv3 ~]$ qstat
+
  hpc-ext-2 pauprun $ squeue
  job-ID prior   name       user        state submit/start at    queue                          slots ja-task-ID
+
            JOBID PARTITION    NAME    USER ST      TIME NODES NODELIST(REASON)
-----------------------------------------------------------------------------------------------------------------
+
            645170  general pauprun   plewis PD       0:00      1 (Priority)
  24747 0.50500 gopaup    plewis      r    01/18/2014 21:30:02 all.q@compute-1-2.local            1
+
The PD under ST (state) means that my job is pending (not yet running). This job goes so fast that you will be lucky to find it in the running state. If you see no jobs listed when you run squeue, it means your job has finished.
This indicates that my job is now running on the node named compute-1-2.
+
  
==== Killing a job using qdel ====
+
==== Killing a job using scancel ====
  
Sometimes it is clear that an analysis is not going to do what you wanted it to. Suppose that just after you press the Enter key to start an analysis, you realize that you forgot to put in a savetrees command in your paup block (so in the end you will not be able to see the results of the search). In such situations, you really want to just kill the job, fix the problem, and then start it up again. Use the qdel command for this. Note that in the output of the qstat command above, my run had a job-ID equal to 24747. I could kill the job like this:
+
Sometimes it is clear that an analysis is not going to do what you wanted it to. Suppose that just after you press the Enter key to start an analysis, you realize that you forgot to put in a savetrees command in your paup block (so in the end you will not be able to see the results of the search). In such situations, you really want to just kill the job, fix the problem, and then start it up again. Use the scancel command for this. Note that in the output of the squeue command above, my run had a job-ID equal to 645170. I could kill the job like this:
  qdel 24747
+
  scancel 645170
SGE will say that it has scheduled the job for deletion, but in practice it kills it almost instantaneously in my experience. Be sure to delete any output files that have already been created before starting your run over again.
+
Be sure to delete any output files that have already been created before starting your run over again.
  
 
==== While PAUP* is running ====
 
==== While PAUP* is running ====
Line 224: Line 244:
 
====  Using Cyberduck to download the log file and the tree file ====
 
====  Using Cyberduck to download the log file and the tree file ====
  
When PAUP* finishes, qstat will no longer list your process. At this point, you need to use Cyberduck to get the log and tree files that were saved back to your local computer. Assuming you left Cyberduck open and connected to the cluster, double-click on the pauprun directory and locate the files <tt>algae.ml.tre</tt> and  <tt>algae.output.txt</tt>. Select these two files and drag them out of the Cyberduck window and drop them on your desktop. After a flurry of activity, Cyberduck should report that the two files were downloaded successfully, at which point you can close the download status window.
+
When PAUP* finishes, squeue will no longer list your process. At this point, you need to use Cyberduck to get the log and tree files that were saved back to your local computer. Assuming you left Cyberduck open and connected to the cluster, double-click on the pauprun directory and locate the files <tt>algae.ml.tre</tt> and  <tt>algae.output.txt</tt>. Select these two files and drag them out of the Cyberduck window and drop them on your desktop. After a flurry of activity, Cyberduck should report that the two files were downloaded successfully, at which point you can close the download status window.
 +
 
 +
==== Using scp to download the log file and the tree file ====
  
==== (Mac users only) Using scp to download the log file and the tree file ====
+
You can also use scp to get the log and tree files that were saved back to your local computer, but, again, if you are happy with Cyberduck you can skip this section. In the Terminal app on your Mac (or the Git for Windows BASH session on your Windows PC), type the following (being careful to separate the final dot character from everything else by a blank space):
 +
scp xfer:pauprun/algae.output.txt .
 +
scp xfer:pauprun/algae.ml.tre .
 +
This assumes you have set up a shortcut: if not, you will need to use the longer version below (being sure to replace <tt>username</tt> with your own user name on the cluster):
 +
scp username@transfer.cam.uchc.edu:pauprun/algae.output.txt .
 +
scp username@transfer.cam.uchc.edu:pauprun/algae.ml.tre .
  
Mac users can also use scp to get the log and tree files that were saved back to your local computer, but, again, if you are happy with Cyberduck you can skip this section. In the Terminal app on your Mac, press Cmd-t to open a new tab, which will create a new session starting in your home directory on your Mac), then type the following (being careful to separate the final dot character from everything else by a blank space):
+
These scp commands copy the files <tt>algae.output.txt</tt> and <tt>algae.ml.tre</tt> to your current directory (this is what the single dot at the end of each line stands for).
cd Desktop
+
scp username@bbcsrv3.biotech.uconn.edu:pauprun/algae.output.txt .
+
scp username@bbcsrv3.biotech.uconn.edu:pauprun/algae.ml.tre .
+
Again, be sure to replace <tt>username</tt> with your own user name on the cluster. The first command makes your current directory the Desktop folder, and the next two commands copy the files <tt>algae.output.txt</tt> and <tt>algae.ml.tre</tt> to your current directory (this is what the single dot at the end of each line stands for).
+
  
 
=== Using FigTree to view tree files ===
 
=== Using FigTree to view tree files ===
Line 254: Line 277:
 
There are many other options that you can discover in FigTree, but one more thing to try today is to save the tree as a PDF file. Once you have the tree looking just the way you want, choose File > Export PDF...
 
There are many other options that you can discover in FigTree, but one more thing to try today is to save the tree as a PDF file. Once you have the tree looking just the way you want, choose File > Export PDF...
  
==== Why both junk.txt and algae.output.txt? ====
+
==== Why have PAUP* create the log file algae.output.txt? ====
  
In your home directory, SGE saved the output that PAUP* normally sends to the console to a file named junk.txt (we specified that it should do this in the gopaup file). I had you name this file junk.txt because you will not need this file after the run: the log command in your paup block ends up saving the same output in the file algae.output.txt. Why did we tell PAUP* to start a log file if SGE was going to save the output anyway? The main reason is that you can view the log file during the run, but you cannot view junk.txt until the run is finished. There will come a day when you have a PAUP* run that has been going for several days and want to know whether it is 10% or 90% finished. At this point you will appreciate being able to view the output file!
+
In your pauprun directory, SLURM saved the output that PAUP* normally sends to the console to a file named slurm-645170.out (your file will have a different number, however). You will not need this file after the run: the log command in your paup block ends up saving the same output in the file algae.output.txt. Why did we tell PAUP* to start a log file if SGE was going to save the output anyway? The main reason is that you can view the log file during the run, but you cannot view slurm-645170.out until the run is finished. There will come a day when you have a PAUP* run that has been going for several days and want to know whether it is 10% or 90% finished. At this point you will appreciate being able to view the output file!
  
==== Delete junk.txt using the rm command ====
+
==== Delete the slurm-xxxx.out file using the rm command ====
  
Because you do not need junk.txt, delete it using the rm command (the -f stands for force; i.e. don't ask if it is ok, just do it!):
+
Because you do not need the slurm-xxxxx.out file (where the xxxx are placeholders for the job number), delete it using the rm command (the -f stands for force; i.e. don't ask if it is ok, just do it!):
 
  cd
 
  cd
rm -f junk.txt
 
You also no longer need the log and tree files because you downloaded them to your local computer using PSFTP:
 
 
  cd pauprun
 
  cd pauprun
 +
rm -f slurm-*.out
 +
You also no longer need the log and tree files because you downloaded them to your local computer using PSFTP:
 
  rm -f algae.ml.tre
 
  rm -f algae.ml.tre
 
  rm -f algae.output.txt
 
  rm -f algae.output.txt
Line 293: Line 316:
 
  man ls
 
  man ls
 
It is important to know how to escape from a man page! The way to get out is to type the letter q. You can page down using Ctrl-f, page up through a man page using Ctrl-b, go to the end using Shift-g and return to the very beginning using 1,Shift-g (that is, type a 1, release it, then type Shift-g). You can also move line by line in a man page using the down and up arrows, and page by page using the PgUp and PgDn keys.
 
It is important to know how to escape from a man page! The way to get out is to type the letter q. You can page down using Ctrl-f, page up through a man page using Ctrl-b, go to the end using Shift-g and return to the very beginning using 1,Shift-g (that is, type a 1, release it, then type Shift-g). You can also move line by line in a man page using the down and up arrows, and page by page using the PgUp and PgDn keys.
 +
 +
==== Xanadu information ====
 +
 +
You can find a lot of great information about the Xanadu cluster at the [https://bioinformatics.uconn.edu/ Computational Biology Core] web site. In particular, take time to look through the first tutorial named "Understanding the UConn Health Cluster (Xanadu)".
  
 
[[Category: Phylogenetics]]
 
[[Category: Phylogenetics]]
 
__NOTOC__
 
__NOTOC__

Latest revision as of 19:56, 16 January 2020

Adiantum.png EEB 5349: Phylogenetics
The goal of this lab exercise is to show you how to log into the Xanadu computer cluster and perform a basic PAUP* analysis.

Using the UConn Xanadu cluster

The UConn Computational Biology Core is part of the Center for Genome Innovation (CGI). We will use the Xanadu computing cluster located at the UConn Health Center for most of the data crunching we will do in this course. By now, you should have an account on the cluster, and today you will learn how to start analyses remotely (i.e. from your laptop), check on their status, and download the results when your analysis is finished.

Obtaining the necessary communications software

You will be using a couple of simple (and free) programs to communicate with the head node of the cluster.

If you use Windows...please scroll down to the Windows section

If you use MacOS 10.x...

SSH

The program ssh will allow you to communicate with the cluster using a protocol known as SSH (Secure Shell). You will use ssh to send commands to the cluster and see the output generated.

Start by opening the Terminal application, which you can find in the Applications/Utilities folder on your hard drive. Using the Terminal program, you can connect to the cluster with the following command:

ssh username@xanadu-submit-ext.cam.uchc.edu

where username should be replaced by your username on the cluster.

You may wish to install iTerm2, which is a terminal program that makes some things easier than Terminal, but the built-in Terminal will work just fine.

Creating a shortcut

If you want to avoid having to type the long command above every time you want to connect to the cluster, it is possible to create a shortcut. You will need to edit the config file in your .ssh directory. Directories with names beginning with a period are not usually shown, but you can open this directory in Finder by typing

 cd
 open .ssh

in your terminal (the initial cd command changes the directory to your default, a.k.a. home, directory).

If you do not see a file named "config" in your ".ssh" director, create an empty config file using the command

 touch config

Open the config file in a text editor such as BBEdit (NOT a word processor such as Microsoft Word or Pages!) and add the following lines (replace plewis with your actual username on xanadu, and replace /Users/plewis with your home directory path on your Mac):

host xanadu
   HostName xanadu-submit-ext.cam.uchc.edu
   User plewis
   IdentityFile /Users/plewis/.ssh/id_rsa
   Cipher blowfish
host xfer
   HostName transfer.cam.uchc.edu
   User plewis
   IdentityFile /Users/plewis/.ssh/id_rsa
   Cipher blowfish

Once you save the config file, you should be able to just type

 ssh xanadu

to login to the xanadu cluster. (The second entry (xfer) will be used when transferring files to and from the cluster.)

SCP/SFTP

An SCP or SFTP client is needed to to transfer files back and forth using the Secure Copy Protocol (SCP) or Secure File Transfer Protocol (SFTP). I will show you how to transfer files using both methods, but for now you should go ahead and install Cyberduck. Cyberduck provides a nice graphical user interface, but you might find that the command line clients let you get your work done faster once you get used to using them.

If you use Windows...

SSH

The program Git for Windows provides a terminal that will allow you to communicate with the cluster using a protocol known as SSH (Secure Shell). We will not actually use the "git" part of "Git for Windows", although it is there in case you need it later. Instead, you will use the Git for Windows bash shell to send commands to the cluster and see the output generated.

Visit the Git for Windows, press the Download button, and install Git for Windows on your system. Once installed, open Git BASH from the All Programs section of the start menu. This will open a terminal running the bash shell (a shell is a program that interprets operating system control commands) on your desktop.

Connect to the cluster with the following command:

ssh username@xanadu-submit-ext.cam.uchc.edu

where username should be replaced by your username on the cluster.

Creating a shortcut

If you want to avoid having to type the long command above every time you want to connect to the cluster, it is possible to create a shortcut. You will need to edit the config file in your .ssh directory. Open (or create, if it does not yet exist) the file named config in a text editor such as NotePad++ (NOT a word processor such as Microsoft Word!) and add the following lines (replace plewis with your actual username on xanadu, and replace /c/Users/Paul Lewis with your home directory path):

host xanadu
   HostName xanadu-submit-ext.cam.uchc.edu
   User plewis
   IdentityFile "/c/Users/Paul Lewis/.ssh/id_rsa"
   Cipher blowfish
host xfer
   HostName transfer.cam.uchc.edu
   User plewis
   IdentityFile "/c/Users/Paul Lewis/.ssh/id_rsa"
   Cipher blowfish

Use the pwd command to find out what your home directory path is, and use double quotes if your home directory path contains embedded spaces (note that I had to use quotes for mine).

Once you save the config file, you should be able to just type

 ssh xanadu

to login to the xanadu cluster. (The second entry (xfer) will be used for transferring files to and from Xanadu.)

SCP/SFTP

An SCP or SFTP client is needed to to transfer files back and forth using the Secure Copy Protocol (SCP) or Secure File Transfer Protocol (SFTP). For now you should go ahead and install Cyberduck. Cyberduck provides a nice graphical user interface, which makes moving files back and forth easy.

Learning enough UNIX to get around

I'm presuming that you do not know a lot of UNIX commands, but even if you are already a UNIX guru, please complete this section anyway because otherwise you will fail to create some files you will need later.

ls command: finding out what is in the present working directory

The ls command lists the files in the present working directory. Try typing just

ls

If you need more details about files than you see here, type

ls -la

instead. This version provides information about file permissions, ownership, size, and last modification date.

pwd command: finding out what directory you are in

Typing

pwd

shows you the full path of the present working directory. The path shown should end with your username, indicating that you are currently in your home directory.

mkdir command: creating a new directory

Typing the following command will create a new directory named pauprun in your home directory:

mkdir pauprun

Use the ls command now to make sure a directory of that name was indeed created.

cd command: leaving the nest and returning home again

The cd command lets you change the present working directory. To move into the newly-created pauprun directory, type

cd pauprun

You can always go back to your home directory (no matter how lost you get!) by typing just cd by itself

cd

If you want to go down one directory level (say from pauprun back down to your home directory), you can specify the parent directory using two dots:

cd ..

Creating run.nex using the nano editor

One way to create a new file, or edit one that already exists, is to use the nano editor. You will now use nano to create a run.nex file containing a paup block. You will later execute this file in PAUP* to perform an analysis.

First use the pwd command to see where you are, then use cd to go into the pauprun directory if you are not already there. Type

nano run.nex

This will open the nano editor, and it should say [ New File ] at the bottom of the window to indicate that the run.nex file does not already exist. Note the menu of the commands along the bottom two rows. Each of these commands is invoked using the Ctrl key with the letter indicated. Thus, ^X Exit indicates that you can use the Ctrl key in combination with the letter X to exit nano.

For now, type the following into the editor:

#nexus

begin paup;
  log file=algae.output.txt start replace flush;
  execute algae.nex;
  set criterion=likelihood autoclose;
  lset nst=2 basefreq=estimate tratio=estimate rates=gamma shape=estimate;
  hsearch swap=none start=stepwise addseq=random nrep=1;
  lscores 1;
  lset basefreq=previous tratio=previous shape=previous;
  hsearch swap=tbr start=1;
  savetrees file=algae.ml.tre brlens;
  log stop;
  quit;
end;

Once you have entered everything, use ^X to exit. Nano will ask if you want to save the modified buffer (a buffer is a predefined amount of computer memory used to store the text you type; the text stored in the buffer will be lost once you exit the program unless you save it to a file on the hard drive), at which point you should press the Y key to answer yes. Nano will now ask you whether you want to use the file name run.nex; this time just press Enter to accept. Nano should now exit and you can use cat to look at the contents of the file you just created:

cat run.nex

Create the gopaup file

Now use nano to create a second file named gopaup in your pauprun directory. To do this, type pwd to make sure you are in the pauprun directory, then type nano gopaup. This file should contain this text:

 #!/bin/bash
 #SBATCH --partition=general
 #SBATCH --qos=general 
 #SBATCH --job-name=pauprun
 cd $HOME/pauprun
 module load paup/4.0a-166
 paup -n run.nex

Using Cyberduck to upload the algae.nex data file

Cyberduck-bookmark-xanadu.png

Download the file algae.nex from here and save it on your hard drive.

Open Cyberduck, choose Bookmark > New Bookmark from the main menu, then fill out the resulting dialog box as shown on the right (except substitute your own user name, of course). Be sure to change the protocol to SFTP (not the default FTP). Click the button to close the dialog box and you should see your bookmark appear at the bottom of the main window. Double click the bookmark to open a connection. You will then be warned that the host key is unknown - choose Allow (and go ahead and check the Always button so you do not need to do this every time.

Once you are in, you will see a listing of the files in your home directory (if you have any). To copy the algae.nex file to the cluster, you need only drag-and-drop it onto the Cyberduck window.

Using scp to upload the algae.nex data file

While you will probably want to do your file transfers with Cyberduck as described above, it is also possible to transfer files using the command line scp client. Read on if you are interested in this option, but feel free to skip this section if you are happy using Cyberduck.

In your terminal, navigate to where you saved the file. If you saved it on the desktop, you can go there by typing cd Desktop.

If you've made a shortcut in your .ssh/config file, you can use the following command to upload the algae.nex file:

 scp algae.nex xfer:

If you have not made a shortcut, use this command instead:

scp algae.nex username@transfer.cam.uchc.edu:

where username should be replaced by your own user name on the cluster. (Don't overlook the colon on the very end of the line!)

A few more UNIX commands

You have now transferred a data file (algae.nex) to the cluster, but it is not in the right place. The algae.nex file is currently in your home directory, whereas the run.nex file is in the pauprun directory. The run.nex file contains a line containing the command execute algae.nex, which means that algae.nex should also be located in the pauprun directory. Use the following commands to ensure that (1) you are in your home directory, and (2) algae.nex is also in your home directory:

 cd $HOME
 ls algae.*

Note the use of a wildcard character (*) in the ls command. This will show you only files that begin with the letters algae followed by a period and any number of other non-whitespace characters. The $HOME is a predefined shell variable that will be replaced with your home directory. It is not necessary in this case - typing cd all by itself would take you to your home directory - but the $HOME variable is good to know about (especially for use in scripts).

mv command: moving or renaming a file

Now use the mv command to move algae.nex to the directory pauprun:

mv algae.nex pauprun

The mv command takes two arguments. The first argument is the name of the directory or file you want to move, whereas the second argument is the destination. The destination could be either a directory (which is true in this case) or a file name. If the directory pauprun did not already exist, mv would have interpreted this as a request to rename algae.nex to the file name pauprun! So, be aware that mv can rename files as well as move them.

cp command: copying a file

The cp command copies files. It leaves the original file in place and makes a copy elsewhere. You could have used this command to get a copy of algae.nex into the directory pauprun:

cp algae.nex pauprun

This would have left the original in your home directory, and made a duplicate of this file in the directory pauprun.

rm command: cleaning up

The rm command removes files. If you had used the cp command to copy algae.nex into the pauprun directory, you could remove the original file using these commands:

cd
rm algae.nex

The first cd command just ensures that the copy you are removing will be the one in your home directory (typing cd by itself acts the same as typing cd $HOME). To delete an entire directory (don't try this now!), you can add the -rf flags. The r flag tells rm to recursively apply the remove command to everything in every subdirectory, while the f flag means force (don't ask whether each file should be deleted in each subdirectory, just do it!):

rm -rf pauprun

The above command would remove everything in the pauprun directory (without asking!), and then remove the pauprun directory itself. I want to stress that this is a particularly dangerous command, so make sure you are not distracted or sleep-deprived when you use it! Unlike the Windows or Mac graphical user interface, files deleted using rm are not moved to the Recycle Bin or Trash, they are just gone. There is no undo for the rm command.

Starting a PAUP* analysis

If you've been following the directions in sequence, you now have two files (algae.nex and run.nex) in your $HOME/pauprun directory on the cluster, whereas the gopaup file should be in $HOME. Use the cd command to make sure you are in your home directory, then the cat command to look at the contents of the gopaup file you created earlier. You should see this:

#!/bin/bash
#SBATCH --partition=general 
#SBATCH --qos=general
#SBATCH --job-name=pauprun
cd $HOME/pauprun
module load paup/4.0a-166
paup -n run.nex

This file will be used by software called SLURM to start your run. SLURM provides a command called sbatch that you will use to submit your analysis. The SLURM sbatch command will look for a core (i.e. processor) on a node (i.e. machine) in the cluster that is currently not being used and will start your analysis on that node. This saves you the effort of looking amongst all nodes in the cluster for a core that is not busy.

Here is an explanation of each of the lines in gopaup:

  • The 1st line specifies the command interpreter to use (just include this in your scripts verbatim).
  • The 2nd, 3rd, and 4th lines begin with #SBATCH and are interpreted as commands by SLURM itself. In this case, the first and second #SBATCH commands tell SLURM to use the general partition (--partition=general) and the general quality of service (--qos=general). You should always include these two lines verbatim. The last #SBATCH line gives a name to your job (--job-name=pauprun). You could change "pauprun" here to something else, but keep your job names short and without embedded spaces or punctuation. The job name will help you identify your run when checking status.
  • The 5th line is simply a cd command that changes the present working directory to the pauprun directory you created earlier. This will ensure that anything saved by PAUP* ends up in this directory rather than in your home directory. Note that $HOME is like a macro that will be expanded to the full path to your home directory.
  • The 6th line informs the system that you want to use a particular version of paup. If you left this line out, the command on the last line might not work at all, or might run an older version of paup. You can get a list of all available modules using the command "module avail"
  • The 7th and last line starts up PAUP* and executes the run.nex file. The -n flag tells PAUP* that no human is going to be listening or answering questions, so it should just use default answers to any questions it needs to ask during the run.

Submitting a job using sbatch

Now you are ready to start the analysis. Type these commands to start your run:

cd
cd pauprun
sbatch gopaup

Checking status using squeue

You can see if your run is still going using the squeue command:

squeue

If it is running, you will see an entry named pauprun. Here is what it looked like for me:

hpc-ext-2 pauprun $ squeue
            JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           645170   general  pauprun   plewis PD       0:00      1 (Priority)

The PD under ST (state) means that my job is pending (not yet running). This job goes so fast that you will be lucky to find it in the running state. If you see no jobs listed when you run squeue, it means your job has finished.

Killing a job using scancel

Sometimes it is clear that an analysis is not going to do what you wanted it to. Suppose that just after you press the Enter key to start an analysis, you realize that you forgot to put in a savetrees command in your paup block (so in the end you will not be able to see the results of the search). In such situations, you really want to just kill the job, fix the problem, and then start it up again. Use the scancel command for this. Note that in the output of the squeue command above, my run had a job-ID equal to 645170. I could kill the job like this:

scancel 645170

Be sure to delete any output files that have already been created before starting your run over again.

While PAUP* is running

While PAUP* is running, you can use cat to look at the log file:

cd pauprun
cat algae.output.txt

Using Cyberduck to download the log file and the tree file

When PAUP* finishes, squeue will no longer list your process. At this point, you need to use Cyberduck to get the log and tree files that were saved back to your local computer. Assuming you left Cyberduck open and connected to the cluster, double-click on the pauprun directory and locate the files algae.ml.tre and algae.output.txt. Select these two files and drag them out of the Cyberduck window and drop them on your desktop. After a flurry of activity, Cyberduck should report that the two files were downloaded successfully, at which point you can close the download status window.

Using scp to download the log file and the tree file

You can also use scp to get the log and tree files that were saved back to your local computer, but, again, if you are happy with Cyberduck you can skip this section. In the Terminal app on your Mac (or the Git for Windows BASH session on your Windows PC), type the following (being careful to separate the final dot character from everything else by a blank space):

scp xfer:pauprun/algae.output.txt .
scp xfer:pauprun/algae.ml.tre .

This assumes you have set up a shortcut: if not, you will need to use the longer version below (being sure to replace username with your own user name on the cluster):

scp username@transfer.cam.uchc.edu:pauprun/algae.output.txt .
scp username@transfer.cam.uchc.edu:pauprun/algae.ml.tre .

These scp commands copy the files algae.output.txt and algae.ml.tre to your current directory (this is what the single dot at the end of each line stands for).

Using FigTree to view tree files

algae.ml.tre file viewed with FigTree

If you do not already have it, download and install the FigTree application on your laptop. FigTree is a Java application, so you will also need to install a Java Runtime Environment (JRE) if you don't already have one (just start FigTree and it will tell you if it cannot find a JRE). Once FigTree is running, choose File > Open from the menu to open the algae.ml.tre file.

Adjusting taxon label font

The first thing you will probably want to do is make the taxon labels larger or change the font. Expand the Tip Labels section on the left and play with the Font Size up/down control. You can also set font details in the preferences, which will save you a lot of time in the future

Line thickness

You can modify the thickness of the lines used by FigTree to draw the edges of the tree by expanding the Appearance tab.

Ladderization

You can ladderize the tree (make it appear to flow one way or the other) by playing with the Order Nodes option in the Trees tab.

Export tree as PDF

There are many other options that you can discover in FigTree, but one more thing to try today is to save the tree as a PDF file. Once you have the tree looking just the way you want, choose File > Export PDF...

Why have PAUP* create the log file algae.output.txt?

In your pauprun directory, SLURM saved the output that PAUP* normally sends to the console to a file named slurm-645170.out (your file will have a different number, however). You will not need this file after the run: the log command in your paup block ends up saving the same output in the file algae.output.txt. Why did we tell PAUP* to start a log file if SGE was going to save the output anyway? The main reason is that you can view the log file during the run, but you cannot view slurm-645170.out until the run is finished. There will come a day when you have a PAUP* run that has been going for several days and want to know whether it is 10% or 90% finished. At this point you will appreciate being able to view the output file!

Delete the slurm-xxxx.out file using the rm command

Because you do not need the slurm-xxxxx.out file (where the xxxx are placeholders for the job number), delete it using the rm command (the -f stands for force; i.e. don't ask if it is ok, just do it!):

cd
cd pauprun
rm -f slurm-*.out

You also no longer need the log and tree files because you downloaded them to your local computer using PSFTP:

rm -f algae.ml.tre
rm -f algae.output.txt

It is a good idea to delete files you no longer need for two reasons:

  • you will later wonder whether you downloaded those files to your local machine and will have to spend time making sure you actually have saved the results locally
  • our cluster only has so much disk space, and thus it is just not possible for everyone to keep every file they ever created

Tips and tricks

Here are some miscellaneous tips and tricks to make your life easier when communicating with the cluster.

Command completion using the tab key

You can often get away with only typing the first few letters of a filename; try pressing the Tab key after the first few letters and the command interpreter will try to complete the thought. For example, cd into the pauprun directory, then type

cat alg<TAB>

If algae.nex is the only file in the directory in which the first three letters are alg, then the command interpreter will type in the rest of the file name for you.

Wildcards

I've already mentioned this tip, but it bears repeating. When using most UNIX commands that accept filenames (e.g. ls, rm, mv, cp), you can place an asterisk inside the filename to stand in for any number of letters. So

ls algae*

will produce output like this

algae.ml.tre    algae.nex   algae.output.txt

Man pages

If you want to learn more options for any of the UNIX commands, you can use the man command to see the manual for that command. For example, here's how to see the manual describing the ls command:

man ls

It is important to know how to escape from a man page! The way to get out is to type the letter q. You can page down using Ctrl-f, page up through a man page using Ctrl-b, go to the end using Shift-g and return to the very beginning using 1,Shift-g (that is, type a 1, release it, then type Shift-g). You can also move line by line in a man page using the down and up arrows, and page by page using the PgUp and PgDn keys.

Xanadu information

You can find a lot of great information about the Xanadu cluster at the Computational Biology Core web site. In particular, take time to look through the first tutorial named "Understanding the UConn Health Cluster (Xanadu)".