1 year ago
#379203
run_the_race
Python Asyncio aiohttp slower in parallel than sequential requests
Using Python 3.8.10, on Ubuntu 20.04
I have tried various snippets from stackoverflow, and the async requets using aiohttp seem to freeze, or take a really really long time.
The last example I tried is this example from twilio:
The only change I made was from 151
requests to 15
:
import aiohttp
import asyncio
import time
start_time = time.time()
async def get_pokemon(session, url):
async with session.get(url) as resp:
pokemon = await resp.json()
return pokemon['name']
async def main():
async with aiohttp.ClientSession() as session:
tasks = []
for number in range(1, 15):
url = f'https://pokeapi.co/api/v2/pokemon/{number}'
tasks.append(asyncio.ensure_future(get_pokemon(session, url)))
original_pokemon = await asyncio.gather(*tasks)
for pokemon in original_pokemon:
print(pokemon)
asyncio.run(main())
print("--- %s seconds ---" % (time.time() - start_time))
But when I run the code it takes a long time at this line:
original_pokemon = await asyncio.gather(*tasks)
and finally (after 3 minutes) prints:
Prints:
bulbasaur
ivysaur
venusaur
charmander
charmeleon
charizard
squirtle
wartortle
blastoise
caterpie
metapod
butterfree
weedle
kakuna
--- 261.69673109054565 seconds ---
If I use the requests lib:
import time
import requests
start_time = time.time()
for number in range(1, 15):
url = f'https://pokeapi.co/api/v2/pokemon/{number}'
response = requests.get(url)
print('requests', response.json()['name'])
print("--- %s seconds ---" % (time.time() - start_time))
requests bulbasaur
requests ivysaur
requests venusaur
requests charmander
requests charmeleon
requests charizard
requests squirtle
requests wartortle
requests blastoise
requests caterpie
requests metapod
requests butterfree
requests weedle
requests kakuna
--- 1.5592358112335205 seconds ---
It only takes 1.5s, as you can see there is nothing wrong with my internet connection. What is causing aiohttp to run so slow?
If I run the async code for 2 requests, then its very fast:
bulbasaur
venusaur
--- 0.0846853256225586 seconds ---```
Then if I run the async code with for 3 requests, then its very slow again, all 3 results are printed together right at the end:
bulbasaur ivysaur venusaur --- 262.15460085868835 seconds ---
I have a 8 core i7 CPU, so I don't understand where the magic number of 2 comes in.
python
python-requests
aiohttp
0 Answers
Your Answer