1 year ago

#388507

test-img

codecodecode

Why this async aiohttp is so slow/How to speed it up

I adapted aiohttp code pretty directly from this post to create a scrip which I am aiming to query my database 40+ times a second so I can benchmark its throttling. This seems reasonable based on the benchmarks in the article for the section of code I adapted, but my actual results are far poorer (barely breaking 4 queries a second).

Here is the code I'm using:

#!/usr/bin/env python3
# -*- coding: iso-8859-15 -*-
import asyncio
import aiohttp
import datetime. 
TOKEN = [REDACTED]
API_URL = [REDACTED]. 

queries = 40
succ_queries = 0
throttled = False

async def query_api(session):
    global throttled, succ_queries
    async with session.get(API_URL, headers={'Authorization': 'Token {}'.format(TOKEN)}) as resp:
        content = await resp.text()
        #print(content)
        if "Request was throttled." in content:
            throttled = True
        succ_queries += 1
        return resp

async def by_aiohttp_concurrency(total: int):
    # code from https://levelup.gitconnected.com/making-concurrent-web-api-requests-in-python-i-recommend-aiohttp-43f2d46ef4da#a209

    async with aiohttp.ClientSession() as session:
        tasks = [] 
        for _ in range(total):
            tasks.append(asyncio.create_task(query_api(session)))

        original_result = await asyncio.gather(*tasks)
        #for res in original_result:
            #print(res)


        
start_time = datetime.datetime.now()
asyncio.run(by_aiohttp_concurrency(queries))
print("--- It took ", (datetime.datetime.now() - start_time), " seconds to run ", succ_queries, " queries ---"). 

As of now, I'm usually only getting 40 queries in around 9 seconds so I'm really hoping someone can help me out in speeding this up.
Strangely enough, when I tested this same script with a URL not on my dev server I get much better speed, but I thought my code as it stands shouldn't be waiting for server response before sending more queries so I must be fundamentally confused on some level

python

async-await

python-asyncio

aiohttp

0 Answers

Your Answer

Accepted video resources