I am trying to create a bash script and add it to my $PATH to run automatically when logging into the server. I am wanting to download FASTQC and trimgalore and add them to my bash script so they don't have to be downloaded every time I log onto the server. Does anyone know how to do this?
2 answers
I'm guessing here... You have downloaded fastqc, trimgalore etc. Now you want to put them in one of the directories listed in the PATH variable so that when you login you can execute e.g. fastqc without specifying the full path to fastqc. Usually, you do this by moving the programs of interest to ~/bin or ~/.local/bin. These directories may already exist and be in $PATH, if not create them and edit ~/.bashrc to add them. But do echo $PATH to see first what you already have.
Having said that, I suggest installing conda and bioconda and let conda manage program installations for you.
The symlink needs to point to a file with the corresponding name: ln -s /home/usr/tools/FastQC/fastqc ~/.local/bin/fastqc. You only need to do this once.
The directory ~/.local/bin needs to exist and be on your path. You can check with echo $PATH. If not, add it to your path with export PATH=~/.local/bin:$PATH. From now on, every executable symlinked with that directory should be executed when typing the name.
PS: If singularity happens to be installed on your system, there would be an alternative solution...
@Matthias Zepper, I have no experience with singularity or docker in part because our IT refuses to install docker on the basis of security concerns. I don't know if these concerns are valid, but either way, depending on something that only the administrator can install is a bit of a red flag for me. Conda environments are not as isolated as docker/singularity containers but you don't need admin rights and settings things up from scratch is quite simple. E.g. install conda/bioconda, if not already there:
curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
Create your project environment and install dependencies listed in requiremenents.txt:
conda create --yes -n my-env
conda activate my-env
conda install -n my-env --yes --file requirements.txt
If you add these instructions to a README file then you are quite "portable". PS: I would use mamba instead of conda for installing dependencies.
@dariober: Fair enough. Indeed, securing Docker on an HPC system and e.g. mitigating privilege escalation is very difficult, so concerns of your IT department are appropriate. This is why I was asking about Singularity, which very actively restricts permissions so that one does not gain access to additional resources while inside the container. It is therefore the preferred platform for running portable containers in traditional HPC environments.
conda/mamba is nice, and I use it a lot on my local machine, but since our HPC cluster has no internet access, pulling software and dependencies from Bioconda doesn't fly. In contrast, building Singularity images or downloading them from Galaxy & Biocontainers, scp'ing them to the cluster and executing them there does work even with regular user privileges.
Since the original poster didn't specify the exact compute environment and privileges, it is hard to tell what solution is "the best". I agree, that modifying the $PATH is the most straightforward, using conda perfectly fine and mentioned running singularity images just as another option.
Thanks for feedback... I had a look at singularity and it seems it can be installed without root and perhaps IT is going to be less concerned (I haven't asked...)
our HPC cluster has no internet access
That must be a major pain! Depending on how much data you process, I would consider buying a decent desktop or a server with Lynux and moving away from the cluster altogether.
Well, we are also processing human genome data, so the lack of network access is one of several provisions against data breaches.
Can you recommend a decent desktop capable of crunching ~420500 core-hours a month? :-P (And a cheap electricity provider...)
This question does not make sense. Programs do not need to be downloaded every time one logs into the server. They are installed one time (if needed) and can be re-used. Are you logging into a server managed by someone else or are you using a computer you are managing?
Why don't you use conda/bioconda to install these programs?
Log in to answer this question.
-
Converting bedgraph file to wig file
written by margo •
-
Extracting fasta sequence in R
written by margo •
-
Using gtf file to get gene positions
written by margo •
-
Running fastqc
written by margo •
-
Bash script "no such file or directory"
written by samc •
-
How to add path of cutadapt in trim_galore perl script?
written by nadiabeg.comsats •
-
Organizing tools/ software
written by shibl_a •
-
Cannot locate the Vcf.pm every time I logging on the server
written by mike229lin •
-
Uploading narrowPeak Annotation Files to display on UCSC Genome Browser
written by Kevin.Y •
-
Error message " library files are not complete" while running the ./runI-TASSER.pl script?
written by Neha shri •
as dariober mentions below: your .bashrc (or .zshrc, .cshrc ... depending on what shell you use) file is the way to go to get this done.
I will not comment on the feasibility/usefulness of what you try to achieve ;)