This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Learning Cluster Computing With Unix / Linux Servers

Is there is best guideline manuals or one line resources for biginner to work on computer with unix / linux servers ....I am new to this area ...I would like to learn how and what software are available .....

unix linux server

what aspect of "cluster computing" are you trying to learn in particular? Do you want to learn how to develop software that can be distributed and communicate over a cluster (like with MPI, or using something like actors/akka), or do you want to know how to submit jobs to a cluster (like using SGE), or if you can use something like hadoop for something, or what?

basically submit jobs, work and manage clusters

This question is clearly off topic, pls goto e.g. to serverfault for pure server/admin related questions. shutting down...

3 answers

Except for one word (cluster) in the title, this seems like a duplicate question. Are you asking about learning to use unix/linux servers? If so, see the above link. If you are asking specifically about using a cluster, maybe you can clarify. Are you trying to learn practices for parallel processing, cluster management, etc? Maybe you can give an example of what you are trying to accomplish.

I have used high-performance clusters for parallel computing at three institutes. While the basic concepts were the same, the way to use them was very different and highly dependent on the scheduler, etc that were set up by admin staff. I currently use the Lawrencium cluster. You can read through their users guide for a general idea of how things work there. But, again, this will be somewhat institution specific and your best bet is to talk to admin/hpc staff for the cluster you plan to use.

I am interested to learn cluster computing...I known learning OS are linux / unix is separate. My meaning I need to learn operating on unix / linux computing in such computing clusters ...thanks

Other than basic UNIX command textbooks like the "Learning ..." series from O'Reilly, I would say SGE is alive and kicking, so it's useful to learn. Hadoop or similar. Amazon cloud also potentially useful. LSF is proprietary but lots of people use it.

One good way of getting a few tips every once in a while is to go to stackoverflow or superuser and add a couple of RSS tags to your RSS feed. You will receive Q&As related to the tags you subscribe to. I would do:

http://superuser.com/questions/tagged/cluster

and

http://stackoverflow.com/questions/tagged/cluster-computing

If you know how to operate in DOS, then UNIX is pretty similar but some of command are phrased differently. It is case sensitive. A shell is basically the user interface in UNIX. It is a program that allows the system to understand your commands.

Some unix commands to run jobs in unix servers:

Fundamental commands:

  • cd # change directory, the hierarchy where each level is separated by a /. The "root" directory is simply called /. So, if I am in / I can type cd u/ck and I will be in /u/ck, ck's home directory, since it will change directories relative to my current path.
    • cd .. # back to one level down
  • ls # This command lists all directories and files contained inside the current directory, when typed alone.
    • ls -a # Lists all the files in a directory, including the normally hidden . files.
    • ls -l # The long format listing includes permissions, owner, size and modification time
  • cp # To copy a file into a file with a different name, or into another directory, we use the cp command.
  • rm # To remove a file we no longer need, we use the rm command
  • mkdir # To create a new directory, use the mkdir command.
  • chmod # Everything you create has permissions for the people who can read, write to, or execute the file or directory you create. The command ls -l lists the permissions of a file. If you want to change who can read a file you use the command chmod.
    • chmod a+r foo.txt # Add read permission for all:
    • chmod g+rw foo.txt # Add read and write permissions for the group:
    • chmod g+rwx foo.txt # Add read, write execution permissions for the group:
    • chmod o-x foo.txt # remove execute permission for others:
  • cls - Alias often used to clear a screen.
  • edit myfile.txt # Runs the editor for myfile.txt.

Logging On to Machines and submitting jobs

log into your account, or log into another machine remotely - The su command allows you to become the "super-user" or another user in a shell. The rsh command creates a remote shell by connecting to the specified computer. The command rlogin creates a remote login session from your terminal to the machine specified. Once again, you can specify a user name using the -l option.

  • at # Schedules a command to be ran at a particular time, such as a print job late at night.
  • at -m 01:35 < atjob # Run the commands listed in the atjob file at 1:35AM, in addition all output that is generated from job mail to the user running the task.
  • at -l # This command will list each of the scheduled jobs
  • at -r 1072250520.a # delete job 1072250520.a
  • atq # lists the user's pending jobs, unless the user is the superuser; in that case, everybody's jobs are listed.
  • atrm # deletes jobs, identified by their job number.
    • atrm 23 # Deletes job 23.
  • kill -s kill 100 -165 # Kills job 1 of uid 165
  • script myfile.txt # Will log all results into the file myfile.txt.
  • top - Display Linux tasks.
  • history - Display the history of commands typed.
  • host - DNS lookup utility.
  • hostid - Prints the numeric identifier for the current host

Run specific programs

  • perl -w test.pl arg1 arg2 arg3 # run perl with three arguments (arg1, arg2, arg3)

or

#!/usr/bin/perl -w # you can set execute permission to the perl script
test.pl arg1 arg2 arg2

Run R

my script file, saved as myscript.R

Revo CMD BATCH myscript.R myscript.Rout

For details:

Log in to answer this question.