92 lines
4.9 KiB
Org Mode
92 lines
4.9 KiB
Org Mode
---
|
|
title: Some Python asyncio disambiguation
|
|
author: Chris Hodapp
|
|
date: March 9, 2018
|
|
tags: technobabble
|
|
---
|
|
|
|
Recently I needed to work a little more in-depth with [[https://docs.python.org/3/library/asyncio.html][asyncio]] in
|
|
Python 3.x. While some people (including me) might scoff at this
|
|
because cooperative threading is a model that's fresh out of the '90s
|
|
and because Python /still/ has the [[https://wiki.python.org/moin/GlobalInterpreterLock][GIL]], it is still preferable to
|
|
manually writing code in [[https://en.wikipedia.org/wiki/Continuation-passing_style][continuation-passing-style]] (that's all
|
|
callbacks are), and last time I had to write that many callbacks, I
|
|
hated it enough that I wrote my own [[https://github.com/HaskellEmbedded/ion][EDSL]] to avoid it. But I digress.
|
|
|
|
I found the [[https://pymotw.com/3/concurrency.html][Concurrency with Processes, Threads, and Coroutines]]
|
|
tutorials to be approachable and thorough, and I highly recommend
|
|
them.
|
|
|
|
However, I still had a few stumbling blocks in understanding, and
|
|
below I give some notes I wrote to check my understanding. I put
|
|
together a table to try to classify what method to use in different
|
|
circumstances. As I use it here, calling "now" means turning control
|
|
over to some other code, whereas calling "whenever" means retaining
|
|
control but queuing up some code to be run in the background
|
|
asychronously (as much as possible).
|
|
|
|
| Call from | Call to | When | How |
|
|
|-----------+-----------+----------+-----------------------------|
|
|
| Either | Function | Now | Normal function call |
|
|
| Function | Coroutine | Now | `.run_*` in event loop |
|
|
| Coroutine | Coroutine | Now | `await` |
|
|
| Either | Function | Whenever | Event loop `.call_*()` |
|
|
| Either | Coroutine | Whenever | Event loop `.create_task()` |
|
|
| | | | `asyncio.ensure_future()` |
|
|
|
|
* Futures & Coroutines
|
|
|
|
The documentation was also sometimes vague on the relation between
|
|
coroutines and futures. My summary on what I figured out is below.
|
|
|
|
** Coroutines and Futures are *mostly* independent.
|
|
|
|
It just happens that both allow you to call things asychronously.
|
|
However, you can use coroutines/asyncio without ever touching a
|
|
Future. Likewise, you can use a Future without ever touching a
|
|
coroutine or asyncio. Note that its `.result()` call isn't a
|
|
coroutine.
|
|
|
|
** They can still encapsulate each other.
|
|
|
|
A coroutine can encapsulate a Future simply by `await`ing it.
|
|
|
|
A Future can encapsulate a coroutine with [[https://docs.python.org/3/library/asyncio-task.html#asyncio.ensure_future][asyncio.ensure_future()]] or
|
|
the event loop's [[https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.create_task][.create_task()]].
|
|
|
|
** Futures can implement asychronicity(?) differently
|
|
|
|
The ability to make a Future from a coroutine was mentioned above;
|
|
that's [[https://docs.python.org/3/library/asyncio-task.html#task][asyncio.Task]], an implementation of [[https://docs.python.org/3/library/asyncio-task.html#future][asyncio.Future]], but it's not
|
|
the only way to make a Future.
|
|
|
|
[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future][concurrent.futures.Future]] is another mostly-compatible way. Its
|
|
[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor][ThreadPoolExecutor]] provides Futures based on separate threads, and its
|
|
[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor][ProcessPoolExecutor]] provides Futures based on separate processes.
|
|
|
|
** Futures are always paired with some running context.
|
|
|
|
That is, a Future is already "started" - running, or scheduled to run,
|
|
or already ran, or something along those lines, and this is why it has
|
|
semantics for things like cancellation.
|
|
|
|
A coroutine by itself is not. The closest analogue is [[https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Handle][asyncio.Handle]]
|
|
which is available only when a coroutine has been scheduled to run.
|
|
|
|
* Other Event Loops
|
|
|
|
[[https://pypi.python.org/pypi/Quamash][Quamash]] implements an asyncio event loop inside of Qt, and I used this
|
|
on a project. I ran into many issues with this combination. Qt's
|
|
juggling of multiple event loops seemed to cause many problems here,
|
|
and I still have some unsolved issues in which calls
|
|
`run_until_complete` cause coroutines to die early with an exception
|
|
because the event loop appears to have died. This came up regularly
|
|
for me because of how often I would want a Qt slot to queue a task in
|
|
the background, and it seems this is an acknowledge [[https://github.com/harvimt/quamash/issues/33][issue]].
|
|
|
|
There is also [[https://github.com/MagicStack/uvloop\][uvloop]]. I have no need for extra performance (nor could
|
|
I really use it alongside Qt), but it's helpful to know about.
|
|
|
|
# Also: What about coroutine generators?
|
|
# Are they anything special?
|