Description
Long story short
I cannot make 40k requests using a semaphore to limit concurrency.
This is based on code from here though actual code is in-lined below.
Expected behaviour
I should be able to make unlimited requests from an aiohttp client to an aiohttp server.
Actual behaviour
Exceptions are thrown. See below.
I was able to make 1 million requests to a locally running httpbin server running under gunicorn.
I dug into it a bit and wondered if ayncio it self didn't like having 40k tasks running, so I faked out the response by modifying aiohttp/client.py with the following code and make ClientSession.get() return this FakeResponseContextManager. It worked just fine when I did this.
So the problem isn't that I'm creating 40,000 or 1,000,000 tasks. Python's asyncio seems to handle that many tasks just fine. Something blows up with this aiohttp client in conjunction with the aiohttp server (again, because serving httpbin test server worked just fine too).
class FakeResponse:
def __init__(self, url):
self._url = url
async def read(self):
await asyncio.sleep(0.01)
return self._url
class FakeResponseContextManager:
def __init__(self, url):
self._url = url
async def __aenter__(self):
await asyncio.sleep(0.01)
return FakeResponse(self._url)
async def __aexit__(self, exc_type, exc, tb):
await asyncio.sleep(0.01)Steps to reproduce
- run the server
- run the client
- see exceptions
server
#!/usr/bin/env python3
from aiohttp import web
async def handle(request):
name = request.match_info.get('name', "Anonymous")
text = "Hello, " + name
return web.Response(body=text.encode('utf-8'))
app = web.Application()
app.router.add_route('GET', '/{name}', handle)
web.run_app(app)client
#!/usr/bin/env python3
import random
import asyncio
from aiohttp import ClientSession
async def fetch(url):
async with ClientSession() as session:
async with session.get(url) as response:
print(url)
return await response.read()
async def bound_fetch(sem, url):
# getter function with semaphore
async with sem:
await fetch(url)
async def run(loop, r):
url = "http://localhost:8080/{}"
tasks = []
# create instance of Semaphore
sem = asyncio.Semaphore(10)
for i in range(r):
# pass Semaphore to every GET request
task = asyncio.ensure_future(bound_fetch(sem, url.format(i)))
tasks.append(task)
responses = asyncio.gather(*tasks)
await responses
number = 40000
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(loop, number))
loop.run_until_complete(future)Your environment
- Linux Mint 18 (based on Ubuntu 16.04)
- Out of the box Python 3.5.2
- aiohttp installed within virtualenv