I have a large global recarray totaling 30GBs of data in a programme running via qsub on a cluster with 256GBs of RAM. I am currently the only user on this cluster so there are no conflicts with the allocation of RAM. When looping over this recarray the system appears to shunt the object to the disc, not keep it held in RAM, thus slowing the loop in excess of 5 fold. I have been looking at using mmap on the object in the following ways and received the following errors.
m = mmap.mmap(myrecarray, 0) MMAP TypeError: only length-1 arrays can be converted to Python scalars m = mmap.mmap(myrecarray.fileno(), 0) AttributeError: record array has no attribute fileno
is it possible to use mmap to hold a recarray object in the RAM, or is this a total misuse of the mmap method or can this only be done for other object types, like strings or files?