I am interested to learn cluster computing...I known learning OS are linux / unix is separate. My meaning I need to learn operating on unix / linux computing in such computing clusters ...thanks
Is there is best guideline manuals or one line resources for biginner to work on computer with unix / linux servers ....I am new to this area ...I would like to learn how and what software are available .....
3 answers
Except for one word (cluster) in the title, this seems like a duplicate question. Are you asking about learning to use unix/linux servers? If so, see the above link. If you are asking specifically about using a cluster, maybe you can clarify. Are you trying to learn practices for parallel processing, cluster management, etc? Maybe you can give an example of what you are trying to accomplish.
I have used high-performance clusters for parallel computing at three institutes. While the basic concepts were the same, the way to use them was very different and highly dependent on the scheduler, etc that were set up by admin staff. I currently use the Lawrencium cluster. You can read through their users guide for a general idea of how things work there. But, again, this will be somewhat institution specific and your best bet is to talk to admin/hpc staff for the cluster you plan to use.
Other than basic UNIX command textbooks like the "Learning ..." series from O'Reilly, I would say SGE is alive and kicking, so it's useful to learn. Hadoop or similar. Amazon cloud also potentially useful. LSF is proprietary but lots of people use it.
One good way of getting a few tips every once in a while is to go to stackoverflow or superuser and add a couple of RSS tags to your RSS feed. You will receive Q&As related to the tags you subscribe to. I would do:
http://superuser.com/questions/tagged/cluster
and
If you know how to operate in DOS, then UNIX is pretty similar but some of command are phrased differently. It is case sensitive. A shell is basically the user interface in UNIX. It is a program that allows the system to understand your commands.
Some unix commands to run jobs in unix servers:
Fundamental commands:
cd# change directory, the hierarchy where each level is separated by a/. The "root" directory is simply called/. So, if I am in / I can typecd u/ckand I will be in/u/ck, ck's home directory, since it will change directories relative to my current path.cd ..# back to one level down
ls# This command lists all directories and files contained inside the current directory, when typed alone.ls -a# Lists all the files in a directory, including the normally hidden.files.ls -l# The long format listing includes permissions, owner, size and modification time
cp# To copy a file into a file with a different name, or into another directory, we use thecpcommand.rm# To remove a file we no longer need, we use thermcommandmkdir# To create a new directory, use the mkdir command.chmod# Everything you create has permissions for the people who can read, write to, or execute the file or directory you create. The commandls -llists the permissions of a file. If you want to change who can read a file you use the commandchmod.chmod a+r foo.txt# Add read permission for all:chmod g+rw foo.txt# Add read and write permissions for the group:chmod g+rwx foo.txt# Add read, write execution permissions for the group:chmod o-x foo.txt# remove execute permission for others:
cls- Alias often used to clear a screen.edit myfile.txt# Runs the editor for myfile.txt.
Logging On to Machines and submitting jobs
log into your account, or log into another machine remotely -
The su command allows you to become the "super-user" or another user in a shell. The rsh command creates a remote shell by connecting to the specified computer. The command rlogin creates a remote login session from your terminal to the machine specified. Once again, you can specify a user name using the -l option.
at# Schedules a command to be ran at a particular time, such as a print job late at night.at -m 01:35 < atjob# Run the commands listed in theatjobfile at 1:35AM, in addition all output that is generated from job mail to the user running the task.at -l# This command will list each of the scheduled jobsat -r 1072250520.a# delete job 1072250520.aatq# lists the user's pending jobs, unless the user is the superuser; in that case, everybody's jobs are listed.atrm# deletes jobs, identified by their job number.atrm 23# Deletes job 23.
kill -s kill 100 -165# Kills job 1 of uid 165script myfile.txt# Will log all results into the file myfile.txt.top- Display Linux tasks.history- Display the history of commands typed.host- DNS lookup utility.hostid- Prints the numeric identifier for the current host
Run specific programs
perl -w test.pl arg1 arg2 arg3# run perl with three arguments (arg1,arg2,arg3)
or
#!/usr/bin/perl -w # you can set execute permission to the perl script
test.pl arg1 arg2 arg2
Run R
my script file, saved as myscript.R
Revo CMD BATCH myscript.R myscript.Rout
For details:
Log in to answer this question.
what aspect of "cluster computing" are you trying to learn in particular? Do you want to learn how to develop software that can be distributed and communicate over a cluster (like with MPI, or using something like actors/akka), or do you want to know how to submit jobs to a cluster (like using SGE), or if you can use something like hadoop for something, or what?
basically submit jobs, work and manage clusters
This question is clearly off topic, pls goto e.g. to serverfault for pure server/admin related questions. shutting down...