This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Question any limit number of posts to download from API?

I tried to download 50000 posts from API by Python requests command. Is it possible? any limit number posts I could download? Thanks.

software error

Post lacks any details what you are talking about. Please add details: Which API, which posts, are you talking about Biostars? What are the errors you get?

Biostars API. I don't get output. Maybe, there is limit to download posts?

#title & comment: n number posts from "https://www.biostars.org/api/post/{id}"
for n in range(400000, 403674):
    n = str(n)
    biostars_web = "https://www.biostars.org/api/post/" + n
    response4 = requests.get(biostars_web)
    response4_json = response4.json()
    for key in response4_json:
        if key == 'title':
           title1 += [response4_json[key]]
        if key == 'xhtml':
           comm = [response4_json[key]]       
           comm = str(comm).replace('[','').replace(']','').rejavascript:document.forms["comment-form"].submit()place('\n','')
           comm = re.sub(clean, '', comm)          
           comment.append(comm)

data_tuples = list(zip(title1, comment))
df = pd.DataFrame(data_tuples, columns = ['title', 'comment'])

#filter with 9 keywords
df_filter = df.comment.str.contains('SRA | BLAST | GeneBank | RefSeek | pubmed | PMC | CDD | PubChem | SRAtoolkit')
df_2 = df[df_filter]
df_2

please at least add one line of time.sleep(n) after each loop....

0 answers

No answers yet.

Log in to answer this question.