1 year ago

#379203

test-img

run_the_race

Python Asyncio aiohttp slower in parallel than sequential requests

Using Python 3.8.10, on Ubuntu 20.04

I have tried various snippets from stackoverflow, and the async requets using aiohttp seem to freeze, or take a really really long time.

The last example I tried is this example from twilio:

The only change I made was from 151 requests to 15:

import aiohttp
import asyncio
import time

start_time = time.time()


async def get_pokemon(session, url):
    async with session.get(url) as resp:
        pokemon = await resp.json()
        return pokemon['name']


async def main():

    async with aiohttp.ClientSession() as session:

        tasks = []
        for number in range(1, 15):
            url = f'https://pokeapi.co/api/v2/pokemon/{number}'
            tasks.append(asyncio.ensure_future(get_pokemon(session, url)))

        original_pokemon = await asyncio.gather(*tasks)
        for pokemon in original_pokemon:
            print(pokemon)

asyncio.run(main())
print("--- %s seconds ---" % (time.time() - start_time))

But when I run the code it takes a long time at this line:

        original_pokemon = await asyncio.gather(*tasks)

and finally (after 3 minutes) prints:

Prints:

bulbasaur
ivysaur
venusaur
charmander
charmeleon
charizard
squirtle
wartortle
blastoise
caterpie
metapod
butterfree
weedle
kakuna
--- 261.69673109054565 seconds ---

If I use the requests lib:

import time
import requests

start_time = time.time()

for number in range(1, 15):
    url = f'https://pokeapi.co/api/v2/pokemon/{number}'
    response = requests.get(url)
    print('requests', response.json()['name'])
print("--- %s seconds ---" % (time.time() - start_time))
requests bulbasaur
requests ivysaur
requests venusaur
requests charmander
requests charmeleon
requests charizard
requests squirtle
requests wartortle
requests blastoise
requests caterpie
requests metapod
requests butterfree
requests weedle
requests kakuna
--- 1.5592358112335205 seconds ---

It only takes 1.5s, as you can see there is nothing wrong with my internet connection. What is causing aiohttp to run so slow?

If I run the async code for 2 requests, then its very fast:

bulbasaur
venusaur
--- 0.0846853256225586 seconds ---```

Then if I run the async code with for 3 requests, then its very slow again, all 3 results are printed together right at the end:

bulbasaur ivysaur venusaur --- 262.15460085868835 seconds ---


I have a 8 core i7 CPU, so I don't understand where the magic number of 2 comes in.

python

python-requests

aiohttp

0 Answers

Your Answer

Accepted video resources