I recommend that you change your code such that it writes the results to a file ,finishes the run closes this file, then you parse out this file.

The truth to this matter is that due to the fact that the reading/writing to the standard output and error is buffered, it is very much possible (although it is rare, but when it happens is maddening) for one stream say the standard output to be waiting for input all the while the program is attempting to write to the other stream (standard error) but it cannot. It is the classical deadlock problem. The subprocess module makes efforts to avoid this scenario, but there are situations where it still happens - especially on Windows a platform that is less covered by open source enthusiasts.

This behavior is independent of programming language, it is just a net effect of running and capturing the output of one tool inside the other (Python). Changing it to a structure that first writes the file then reads it gives you more control and insight into what is actually happening.

I'm still not sure about the error but you can try the following things (I'll update the list if these don't work):

1. Specify the full path to your blastp executable

2. Using subprocess.Popen instead of executing cline() directly

import subprocess, sys

child = subprocess.Popen(str(blastp_cline), shell=(sys.platform!="win32"))

Your first example used the obsolete deprecated legacy blastall command.

The second example uses BLAST+ (the current version of NCBI BLAST), but has a problem:

from Bio.Blast.Applications import NcbiblastpCommandline

my_blast_db = r"C:\Niek\Test2.2.17\Worm\c_elegans.protein.WS200.fasta"
my_blast_file = r"C:\Niek\Test2.2.17\Worm\worm-HD.fasta"

blastp_cline = NcbiblastpCommandline(cmd='blastp', query=my_blast_file, db=my_blast_db, evalue=0.01,
                                                                        outfmt=5, out="wormAll-HD.xml")

print blastp_cline
print blastp_cline()

You've set the cmd=... option, but blastp is the default anyway. If the BLAST+ binaries are not on the $PATH environment variable (which is probably the case on Windows unless you set this up yourself), you'll get the observed error:

WindowsError: [Error 2] The system cannot find the file specified

The fix is to tell Windows where to look - something like this:

from Bio.Blast.Applications import NcbiblastpCommandline

my_blast_db = r"C:\Niek\Test2.2.17\Worm\c_elegans.protein.WS200.fasta"
my_blast_file = r"C:\Niek\Test2.2.17\Worm\worm-HD.fasta"
my_blast_exe = r"C:\Niek\blast-2.2.29+\bin\blastp.exe"

blastp_cline = NcbiblastpCommandline(cmd=my_blast_exe, query=my_blast_file, db=my_blast_db, evalue=0.01,
                                                                        outfmt=5, out="wormAll-HD.xml")

print blastp_cline
print blastp_cline()

That should work... assuming you adjust the my_blast_exe to match wherever you've put BLAST+ on your machine.