Entering edit mode
5.0 years ago
zhangdengwei
▴
210
I am trying to write two files using python, and I encountered the following error:
OSError: [Errno 24] Too many open files: 'unmatch.site_3.txt'.
The main thing made me confused is the code can work well when I input two smaller files with the size of 39M and 61M, however, it died when the input file became as large as 87M. I have sought some solutions but I have no permission to change the Linux system's setting. Is there another way to handle it like through modifying my code. Thanks.
with open(offtarget_output, "w") as fo:
with open(unmatch_output, "w") as fu:
header = ["Chromosome", "strand", "Coordinate", "off-target site", "Mutation type", "Distance score", "Reads"]
print(*header, sep="\t", file=fo)
print(*header, sep="\t", file=fu)
for key, value in sorted(match_dic.items(), key=lambda item:item[1][-1], reverse=True):
if value[4] != "*":
print(value[0], value[4], key, value[1], value[2], value[3], str(int(value[5])), sep="\t", file=fo)
else:
print(value[0], value[4], key, value[1], value[2], value[3], str(int(value[5])), sep="\t", file=fu)
another thing, you can also do (or should do):
it still can't work, and I could not find which code is critical.
You have to post the full script then. Otherwise it’s just guessing.
Are you the only user of the server? Did you already checked the max open files?
How do you build
match_dic? Do you wrap a loop around the code block above (is the3inunmatch.site_3.txta incremental counter)? In case you are, either you generate too many file handles without closing them or something prevents thewithstatement to close them.Thanks! I found the below code can work, but I am still not clear about the cause, the dictionary shouldn't be sorted in the loop?
Hi, I am not an expert in python, but it is clear to me that you are trying to open too many files. Changing ulimits will only conceal symptoms if it helps at all, unless your ulimit is set to unreasonably low values. What I can see is that your code is just an excerpt, so it might well be that you are opening these files in a loop and therefore have many more open files than you expect.
The with statement in Python simplifies exception handling by encapsulating common preparation and clean-up tasks in so-called context managers. This allows common try..except..finally usage patterns to be encapsulated for convenient reuse and reduce the amount of code you need to write for handling different kinds of exceptions. The with statement creates resources within a block . You write your code using the resources within the block. When the block exits the resources are cleanly released regardless of the outcome of the code in the block (that is whether the block exits normally or because of an exception).