Biostar Beta. Not for public use.
So frustrated compiling necessary packages in a linux system
4
Entering edit mode
18 months ago
johnnytam100 • 100

I have just moved to a new lab and have to deal with installing all necessary packages for bioinformatics in a linux system again.

As I have experienced in the previous lab, compiling packages is a pain as error always appear and prerequisites seem endless.

My obstacles:

1) I do not have root permission

2) I cannot use sudo or yum or even git etc. (as either I do not have root permission or they were not installed)

I want to learn how to compile everything with ease in a linux system.

May I have not only the 'how-tos' but also some rule of thumbs to follow, starting from scratch?

The linux system in the new lab is a CentOS7 (centos-release-7-5.1804.4.el7.centos.x86_64)

Thanks a lot!

0
Entering edit mode

Between conda, docker and brew, you can install pretty much everything without root permissions.

1
Entering edit mode

linuxbrew/homebrew at least need git installed

0
Entering edit mode

I have installed bioconda right now ;) Do you think I need the other two?

0
Entering edit mode

Docker might be useful too. The trend seems to be for bioinformatics packages to be in those two managers lately. Brew is not as popular as it once was.

Conda will have you covered for a significant number of tools alone though.

0
Entering edit mode

Alternatively, you may consider installing GVL or BioLinux as your base install image, which comes with a significant amount of software preconfigured.

0
Entering edit mode

How do you compare BioLinux with Ubuntu? I am planning to build my own linux machine and now I'm choosing between the two!

0
Entering edit mode

BioLinux is Ubuntu, it just ships with a lot of preinstalled bioinformatics packages. I’m not sure whether the project is still ongoing/supported, but there are images available nonetheless.

To my knowledge you shouldn’t lose any of the ‘ubuntu Experience’ on BioLinux

0
Entering edit mode

My preferred option as a non root user is to find the programs at github, download, tar unzip and configure, make.

KK

0
Entering edit mode

I did that actually... maybe the core of my problem is I got errors saying prerequisites are not installed, which needs root permission.

7
Entering edit mode
15 months ago
China

you need bioconda: https://bioconda.github.io/

0
Entering edit mode

+1 for the bioconda suggestion. It really has made my work easier.

0
Entering edit mode

Many thanks! I have installed bioconda now! Great package!

5
Entering edit mode
16 months ago
WCIP | Glasgow | UK

I'm not sure mine is a good advice... I usually install programs under my home directory. For programs whose installation is based on make, usually this works:

./configure --prefix=$HOME make make install ..etc  For python packages, usually this works: pip install --user <my package>  For R packages, install.packages(...) prompts you to install in a user directory if the default one is read-only. And so on... I installed all sorts of things in my home directory, postgres, git, python etc... But yes, be prepared for some frustration and a lot of scanning of StackOverflow posts (but it's a good learning experience anyway)! ADD COMMENTlink 0 Entering edit mode I guess, when you are not root, you cannot do make install I guess. But still the programs would have been compiled under that user, so you can actually use them by giving the full path of the program. However if you do not want to run the programs like that, you can export the path. KK ADD REPLYlink 1 Entering edit mode Hi- the point of --prefix=/my/dir is to instruct make install to put files and dependencies under /my/dir, where /my/dir is a directory where you have full permissions, like your home directory. So make install shouldn't complain. If then $HOME/bin is in your PATH (and it should be, if not add it) executables will be available from anywhere you call them.

The disadvantage of installing stuff in your home, in my experience, is that you may end up with a mixture of system-wide and user dependencies which can cause headaches to troubleshoot (I do have some comic-horror stories to tell...) but as I said, it may be a good learning exercise in itself.

1
Entering edit mode

On most Linux distributions, if you make a bin directory under ~, it will be automatically added to PATH for you ;)

0
Entering edit mode

You shouldn’t need root to use make.

0
Entering edit mode

Thanks for the comment! I always got errors stating prerequisites are required. Unfortunately, I cannot install those prerequisites without root permission...

3
Entering edit mode
23 months ago
Saudi Arabia

Do you have your own linux box where you are root.?

If yes I would recommend learning singularity. Build a singularity image on your own linux box and then copy and run it on lab machine . https://www.sylabs.io/singularity

0
Entering edit mode

Second this, and add that with VM you can run Linux OS on any local machine and so have access to root. Then test/build your Singularity recipe there, upload and pull from Singularity-Hub as user. Such a good system!

1
Entering edit mode

May I know what is special with singularity? Thanks!

1
Entering edit mode

Singularity is a 'container', which is basically a portable computing environment. I think of it like a virtual machine, but at a software level. Containers are built based on 'recipes' which define the environment, including setup of all libraries, software, tools, reference data etc. just like you are installing to your server as root. Containers can be hosted on Singularity-Hub from where you can 'pull' to your local machine as user (no root required). And once defined, and used on a project, the container is available to anyone so that your workflow is completely reproducible, down to the exact compute environment. That's the really special part IMO.

NB building containers on server requires root (pulling from hub does not). Install of Singularity (available for *nix, OsX, Win) to the server also requires root, but the case for a sysadmin to install is very straightforward as it basically cuts out a lot of work for them! Also NB that you can use (bio)conda in a container recipe.

Once you have pulled/built the container on your server, you have an image that you can specify to execute jobs inside. You can shell into it like it is another server. And so your server doesn't have e.g. aligners like BWA or STAR installed, your container image does, and you therefore align your fastq in the container, with all output going back out of the container to your server.

1
Entering edit mode
23 months ago
Saudi Arabia

As we speak few days ago, singularity 3.0 came out and it has remote builder option. You do not need to be root or even own a linux box to build singularity image on cloud . https://cloud.sylabs.io/builder Is this the magic wand that just allows bioinformatician to build/test/run their pipelines without having to worry about sys admins :)

0
Entering edit mode

How do you compare singularity with other linux OS e.g. ubuntu and biolinux? I have a spare computer and would really like to build my own linux machine. I am choosing which OS to install.