diff --git a/drafts/2018-03-09-python-asyncio.org b/drafts/2018-03-09-python-asyncio.org new file mode 100644 index 0000000..6bbbf0d --- /dev/null +++ b/drafts/2018-03-09-python-asyncio.org @@ -0,0 +1,91 @@ +--- +title: Some Python asyncio disambiguation +author: Chris Hodapp +date: March 9, 2018 +tags: technobabble +--- + +Recently I needed to work a little more in-depth with [[https://docs.python.org/3/library/asyncio.html][asyncio]] in +Python 3.x. While some people (including me) might scoff at this +because cooperative threading is a model that's fresh out of the '90s +and because Python /still/ has the [[https://wiki.python.org/moin/GlobalInterpreterLock][GIL]], it is still preferable to +manually writing code in [[https://en.wikipedia.org/wiki/Continuation-passing_style][continuation-passing-style]] (that's all +callbacks are), and last time I had to write that many callbacks, I +hated it enough that I wrote my own [[https://github.com/HaskellEmbedded/ion][EDSL]] to avoid it. But I digress. + +I found the [[https://pymotw.com/3/concurrency.html][Concurrency with Processes, Threads, and Coroutines]] +tutorials to be approachable and thorough, and I highly recommend +them. + +However, I still had a few stumbling blocks in understanding, and +below I give some notes I wrote to check my understanding. I put +together a table to try to classify what method to use in different +circumstances. As I use it here, calling "now" means turning control +over to some other code, whereas calling "whenever" means retaining +control but queuing up some code to be run in the background +asychronously (as much as possible). + +| Call from | Call to | When | How | +|-----------+-----------+----------+-----------------------------| +| Either | Function | Now | Normal function call | +| Function | Coroutine | Now | `.run_*` in event loop | +| Coroutine | Coroutine | Now | `await` | +| Either | Function | Whenever | Event loop `.call_*()` | +| Either | Coroutine | Whenever | Event loop `.create_task()` | +| | | | `asyncio.ensure_future()` | + +* Futures & Coroutines + +The documentation was also sometimes vague on the relation between +coroutines and futures. My summary on what I figured out is below. + +** Coroutines and Futures are *mostly* independent. + +It just happens that both allow you to call things asychronously. +However, you can use coroutines/asyncio without ever touching a +Future. Likewise, you can use a Future without ever touching a +coroutine or asyncio. Note that its `.result()` call isn't a +coroutine. + +** They can still encapsulate each other. + +A coroutine can encapsulate a Future simply by `await`ing it. + +A Future can encapsulate a coroutine with [[https://docs.python.org/3/library/asyncio-task.html#asyncio.ensure_future][asyncio.ensure_future()]] or +the event loop's [[https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.create_task][.create_task()]]. + +** Futures can implement asychronicity(?) differently + +The ability to make a Future from a coroutine was mentioned above; +that's [[https://docs.python.org/3/library/asyncio-task.html#task][asyncio.Task]], an implementation of [[https://docs.python.org/3/library/asyncio-task.html#future][asyncio.Future]], but it's not +the only way to make a Future. + +[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future][concurrent.futures.Future]] is another mostly-compatible way. Its +[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor][ThreadPoolExecutor]] provides Futures based on separate threads, and its +[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor][ProcessPoolExecutor]] provides Futures based on separate processes. + +** Futures are always paired with some running context. + +That is, a Future is already "started" - running, or scheduled to run, +or already ran, or something along those lines, and this is why it has +semantics for things like cancellation. + +A coroutine by itself is not. The closest analogue is [[https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Handle][asyncio.Handle]] +which is available only when a coroutine has been scheduled to run. + +* Other Event Loops + +[[https://pypi.python.org/pypi/Quamash][Quamash]] implements an asyncio event loop inside of Qt, and I used this +on a project. I ran into many issues with this combination. Qt's +juggling of multiple event loops seemed to cause many problems here, +and I still have some unsolved issues in which calls +`run_until_complete` cause coroutines to die early with an exception +because the event loop appears to have died. This came up regularly +for me because of how often I would want a Qt slot to queue a task in +the background, and it seems this is an acknowledge [[https://github.com/harvimt/quamash/issues/33][issue]]. + +There is also [[https://github.com/MagicStack/uvloop\][uvloop]]. I have no need for extra performance (nor could +I really use it alongside Qt), but it's helpful to know about. + +# Also: What about coroutine generators? +# Are they anything special?