This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to install pip on HPC server

I am working on HPC server where already different version of python installed.

I need to use pip to install some bioinformatics software.

But When I am trying to load latest version of python and after that when type python version it always showing old python version so please suggest How I can install pip first and after that how should I use it to install other software:

[3052751@login1(alces) ~]$ python --version
Python 2.6.6
[3052751@login1(alces) ~]$ module load apps/python3
Unable to load apps/python3/3.4.3/gcc-4.4.7 -- alternative module (apps/python3/3.6.4/gcc-4.4.7) already loaded.
[3052771@login1(alces) ~]$ python --version
Python 2.6.6
[3052751@login1(alces) ~]$ pip --version
-bash: pip: command not found
software error
• 9,590 views • link
written by Bioinfonext Bioinfonext

Your module load command had issues and anyway you should try pip3.

• 0 views • link
written by Devon Ryan Devon Ryan

What do you get with module avail python? This command should list all python modules available on your cluster.

• 0 views • link
written by GenoMax GenoMax 
[3052751@login1(alces) ~]$ module avail python

---  /opt/gridware/etc/el6/modules  ---

---  /opt/apps/etc/modules  ---

  python/gcc-4.4.7/2.7.9

---  /opt/service/etc/modules  ---

[3052751@login1(alces) ~]$
• 0 views • link
updated by GenoMax GenoMax • written by Bioinfonext Bioinfonext

What do you get if you do following by running in your $HOME.

find . -type f -iname "*htseq*"
• 0 views • link
written by GenoMax GenoMax

Hi genomax:

[3052751@login2(alces) ~]$ find . -type f -iname "*htseq*"

./.python-eggs/HTSeq-0.6.1p1-py2.7-linux-x86_64.egg-tmp/HTSeq/_HTSeq.so

./gridware/share/python/3.4.3/lib/python3.6/site-packages/HTSeq/_HTSeq.cpython-36m-x86_64-linux-gnu.so

./gridware/share/python/3.4.3/lib/python3.6/site-packages/HTSeq/__pycache__/_HTSeq_internal.cpython-36.pyc

./gridware/share/python/3.4.3/lib/python3.6/site-packages/HTSeq/_HTSeq_internal.py

./gridware/share/python/3.4.3/bin/htseq-qa

./gridware/share/python/3.4.3/bin/htseq-count

./.local/lib/python3.6/site-packages/HTSeq/_HTSeq.cpython-36m-x86_64-linux-gnu.so

./.local/lib/python3.6/site-packages/HTSeq/__pycache__/_HTSeq_internal.cpython-36.pyc

./.local/lib/python3.6/site-packages/HTSeq/_HTSeq_internal.py

./.local/bin/htseq-qa

./.local/bin/htseq-count
• 0 views • link
updated by GenoMax GenoMax • written by Bioinfonext Bioinfonext

Well the programs you are looking for are right there. In two places.

./.local/bin/htseq-count

and

./gridware/share/python/3.4.3/bin/htseq-count

If your $HOME is not available on all cluster nodes then you should use ./gridware/share/python/3.4.3/bin/htseq-count.

• 0 views • link
written by GenoMax GenoMax 
Hi genomax:

Older version of htseq is already install on HPC server: that I can run by using below command but How should I run recent one that I have installed:

older one:

[3052751@login2(alces) ~]$ module add apps/htseq

[3052751@login2(alces) ~]$ python -m HTSeq.scripts.count

Usage: count.py [options] alignment_file gff_file

This script takes an alignment file in SAM/BAM format and a feature file in
GFF format and calculates for each feature the number of reads mapping to it.
See http://www-huber.embl.de/users/anders/HTSeq/doc/count.html for details.

Options:
  -h, --help            show this help message and exit

  -f SAMTYPE, --format=SAMTYPE

                        type of <alignment_file> data, either 'sam' or 'bam
'
                        (default: sam)

  -r ORDER, --order=ORDER
• 0 views • link
written by Bioinfonext Bioinfonext 
latest one:

[3052751@login2(alces) ~]$ python -m /gridware/share/python/3.4.3/bin/htseq-count

/opt/gridware/pkg/el6/apps/python/2.7.8/gcc-4.4.7/bin/python: Import by filename is not supported.
• 0 views • link
written by Bioinfonext Bioinfonext 
module purge apps/htseq

To remove the old module you have already loaded.

Then run (from the directory where you ran the find command)

./.local/bin/htseq-count -h

or

./gridware/share/python/3.4.3/bin/htseq-count -h

You should not need to add python3 at beginning.

• 0 views • link
written by GenoMax GenoMax

I run find commad from the same location:

[3052751@login2(alces) ~]$ /gridware/share/python/3.4.3/bin/htseq-count -h

-bash: /gridware/share/python/3.4.3/bin/htseq-count: No such file or directory

[3052751@login2(alces) ~]$ ./.local/bin/htseq-count -h

/opt/gridware/pkg/el6/apps/python3/3.6.4/gcc-4.4.7/bin/python3.6: error while loading shared libraries: 

libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

[3052751@login2(alces) ~]$
• 0 views • link
written by Bioinfonext Bioinfonext

What modules are currently loaded?

module list
• 0 views • link
written by GenoMax GenoMax

[

3052751@login2(alces) ~]$ module list

No Modulefiles Currently Loaded.
• 0 views • link
written by Bioinfonext Bioinfonext

Do I need to load python3?

• 0 views • link
written by Bioinfonext Bioinfonext

Yes you should try that.

• 0 views • link
written by GenoMax GenoMax 
[3052771@login2(alces) ~]$ module load apps/python3

[3052751@login2(alces) ~]$ ./gridware/share/python/3.4.3/bin/htseq-count -h

/opt/gridware/pkg/el6/apps/python3/3.6.4/gcc-4.4.7/bin/python3.6: error while lo                                          

ading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No                                           such file or directory
[3052751@login2(alces) ~]$
• 0 views • link
written by Bioinfonext Bioinfonext

Something is set up strangely on your cluster. At this point you are going to need some local help to figure out what is going on.

I have run out of things to try :-(

• 0 views • link
written by GenoMax GenoMax

Hi genomax,

thanks for your time and so much effort, surely I am going ask to admin here and let you know if something comes out.

• 0 views • link
written by Bioinfonext Bioinfonext

1 answer

It appears that you already have python v.3.6.4 module loaded. Normally python3 is accessed by doing python3 -V. pip should alreayd be installed there (try pip3 as @Devon suggested). If not use the directions here.

Ideally you should ask the HPC admins to install this so all users can benefit.

• 0 views • link
written by GenoMax GenoMax

Hi genomax,

Now it is showing python3

[3052771@login1(alces) ~]$ python3 -V

Python 3.6.4

How I can check pip already install or not?

Yes it is showing pip3:

[3052771@login1(alces) ~]$ pip3 --version

pip 9.0.1 from /opt/gridware/pkg/el6/apps/python3/3.6.4/gcc-4.4.7/lib/python3.6/site-packages (python 3.6)
• 0 views • link
written by Bioinfonext Bioinfonext

Now I am trying to install latest version of HTseq but I am not able to found in current directory:where it is install ?

[

3052771@login1(alces) ~]$ pip3 install HTSeq

Collecting HTSeq
  Downloading HTSeq-0.9.1-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
    100% |################################| 1.1MB 866kB/s
Collecting pysam>=0.9.0 (from HTSeq)
  Downloading pysam-0.13-cp36-cp36m-manylinux1_x86_64.whl (9.3MB)
    100% |################################| 9.3MB 19kB/s
Collecting numpy (from HTSeq)
  Downloading numpy-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (17.2MB)
    100% |################################| 17.2MB 30kB/s
Installing collected packages: pysam, numpy, HTSeq
Successfully installed HTSeq-0.9.1 numpy-1.14.0 pysam-0.13

[3052771@login1(alces) ~]$ ls

clusterware-ssh-keygen.log  gridware  hisat2-2.1.0  ncbi  NGS_Tools
• 0 views • link
written by Bioinfonext Bioinfonext

I think pip installs into $HOME/.local/bin. Check there.

• 0 views • link
written by GenoMax GenoMax

I just try this:

[3052771@login1(alces) ~]$ pwd

/users/3052751

[3052771@login1(alces) ~]$ cd /home/local/bin

-bash: cd: /home/local/bin: No such file or directory
• 0 views • link
written by Bioinfonext Bioinfonext 
[3052751@login1(alces) ~]$ ls ~/.local/bin

ls: cannot access /users/3052751/.local/bin: No such file or directory
• 0 views • link
written by Bioinfonext Bioinfonext 
[3052751@login1(alces) ~]$ ls ~/local/bin

ls: cannot access /users/3052751/local/bin: No such file or directory
• 0 views • link
written by Bioinfonext Bioinfonext

Check the period before local .local.

• 0 views • link
written by GenoMax GenoMax

I'm a bit surprised that that worked without --user, though perhaps your admins modified things to make it the default.

• 0 views • link
written by Devon Ryan Devon Ryan

how can I try to install to it again with path location?

• 0 views • link
written by Bioinfonext Bioinfonext

I think there's a --force option, though usually specifying either --user or --prefix should do the trick.

• 0 views • link
written by Devon Ryan Devon Ryan 
[3052751@login1(alces) ~]$ pip3 install HTSeq --user /users/3052771/

Directory '/users/3052751/' is not installable. File 'setup.py' not found.
• 0 views • link
written by Bioinfonext Bioinfonext

You can't just randomly try combinations of things. --user takes no parameters, which is why you're getting that results. I'm not going to provide you with specific commands to run, I expect you to be able to use the information provided to construct reasonable commands.

• 0 views • link
written by Devon Ryan Devon Ryan

Hi Devon,

Thanks for your all help: I try this also but not finding the HTseq:

[3052751@login1(alces) ~]$ pip3 install HTSeq --user

Collecting HTSeq
  Using cached HTSeq-0.9.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy (from HTSeq)
  Using cached numpy-1.14.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pysam>=0.9.0 (from HTSeq)
  Using cached pysam-0.13-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy, pysam, HTSeq
Successfully installed HTSeq-0.9.1 numpy-1.14.0 pysam-0.13
[3052751@login1(alces) ~]$ ls
clusterware-ssh-keygen.log  gridware  hisat2-2.1.0  ncbi  NGS_Tools
• 0 views • link
written by Bioinfonext Bioinfonext

You should ask your admins where the default location for pip installs is set to? Sounds like it is not in $HOME/.local/bin. You could also run find . --name HTseq -print to see if that locates the program.

• 0 views • link
written by GenoMax GenoMax

Thanks I will ask to admin about this.

Thaks a lot for your time and prompt response.

• 0 views • link
written by Bioinfonext Bioinfonext

Log in to answer this question.

More posts like this