1 year ago
#388507
codecodecode
Why this async aiohttp is so slow/How to speed it up
I adapted aiohttp code pretty directly from this post to create a scrip which I am aiming to query my database 40+ times a second so I can benchmark its throttling. This seems reasonable based on the benchmarks in the article for the section of code I adapted, but my actual results are far poorer (barely breaking 4 queries a second).
Here is the code I'm using:
#!/usr/bin/env python3
# -*- coding: iso-8859-15 -*-
import asyncio
import aiohttp
import datetime.
TOKEN = [REDACTED]
API_URL = [REDACTED].
queries = 40
succ_queries = 0
throttled = False
async def query_api(session):
global throttled, succ_queries
async with session.get(API_URL, headers={'Authorization': 'Token {}'.format(TOKEN)}) as resp:
content = await resp.text()
#print(content)
if "Request was throttled." in content:
throttled = True
succ_queries += 1
return resp
async def by_aiohttp_concurrency(total: int):
# code from https://levelup.gitconnected.com/making-concurrent-web-api-requests-in-python-i-recommend-aiohttp-43f2d46ef4da#a209
async with aiohttp.ClientSession() as session:
tasks = []
for _ in range(total):
tasks.append(asyncio.create_task(query_api(session)))
original_result = await asyncio.gather(*tasks)
#for res in original_result:
#print(res)
start_time = datetime.datetime.now()
asyncio.run(by_aiohttp_concurrency(queries))
print("--- It took ", (datetime.datetime.now() - start_time), " seconds to run ", succ_queries, " queries ---").
As of now, I'm usually only getting 40 queries in around 9 seconds so I'm really hoping someone can help me out in speeding this up.
Strangely enough, when I tested this same script with a URL not on my dev server I get much better speed, but I thought my code as it stands shouldn't be waiting for server response before sending more queries so I must be fundamentally confused on some level
python
async-await
python-asyncio
aiohttp
0 Answers
Your Answer