This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Inserting millions of records into MySQL database using Python

Hi,

I have a txt file with about 100 million records (numbers). I am reading this file in Python and inserting it to MySQL database using simple insert statement from python. But its taking very long and looks like the script wouldn't ever finish. What would be the optimal way to carry out this process? The script is using less than 12% of memory and 10 to 15% of CPU.

Any suggestions to handle such large data and insert it efficiently into database, would be greatly appreciated.

Thanks

mysql python
• 4,351 views • link
updated by Ram Ram • written by Prasad • Prasad

This seems a question more appropriate for StackOverflow, and indeed, there are loads of suggestions: see here, here, here and here.

• 0 views • link
updated by Ram Ram • written by h.mon h.mon

Thanks for the links. I cannot use Load Data Infile as my input file needs some particular processing before inserting, it cannot be inserted into database as it is. I tried the other things but nothing helps.

Thanks for suggesting to post on Stackoverflow, I just posted it there too. The file I have is made of Gene IDs so I thought of posting here.

• 0 views • link
updated by Ram Ram • written by Prasad • Prasad

Hello Prasad!

We believe that this post does not fit the main topic of this site.

Like h.mon says, this belongs on stackoverflow

For this reason we have closed your question. This allows us to keep the site focused on the topics that the community can help with.

If you disagree please tell us why in a reply below, we'll be happy to talk about it.

Cheers!

• 0 views • link
written by Ram Ram

0 answers

No answers yet.

Log in to answer this question.

More posts like this