---
title: Some Python asyncio disambiguation
author: Chris Hodapp
date: March 9, 2018
tags: technobabble
---

Recently I needed to work a little more in-depth with [[https://docs.python.org/3/library/asyncio.html][asyncio]] in
Python 3.x.  While some people (including me) might scoff at this
because cooperative threading is a model that's fresh out of the '90s
and because Python /still/ has the [[https://wiki.python.org/moin/GlobalInterpreterLock][GIL]], it is still preferable to
manually writing code in [[https://en.wikipedia.org/wiki/Continuation-passing_style][continuation-passing-style]] (that's all
callbacks are), and last time I had to write that many callbacks, I
hated it enough that I wrote my own [[https://github.com/HaskellEmbedded/ion][EDSL]] to avoid it.  But I digress.

I found the [[https://pymotw.com/3/concurrency.html][Concurrency with Processes, Threads, and Coroutines]]
tutorials to be approachable and thorough, and I highly recommend
them.

However, I still had a few stumbling blocks in understanding, and
below I give some notes I wrote to check my understanding.  I put
together a table to try to classify what method to use in different
circumstances.  As I use it here, calling "now" means turning control
over to some other code, whereas calling "whenever" means retaining
control but queuing up some code to be run in the background
asychronously (as much as possible).

| Call from | Call to   | When     | How                         |
|-----------+-----------+----------+-----------------------------|
| Either    | Function  | Now      | Normal function call        |
| Function  | Coroutine | Now      | `.run_*` in event loop      |
| Coroutine | Coroutine | Now      | `await`                     |
| Either    | Function  | Whenever | Event loop `.call_*()`      |
| Either    | Coroutine | Whenever | Event loop `.create_task()` |
|           |           |          | `asyncio.ensure_future()`   |

* Futures & Coroutines

The documentation was also sometimes vague on the relation between
coroutines and futures.  My summary on what I figured out is below.

** Coroutines and Futures are *mostly* independent.

It just happens that both allow you to call things asychronously.
However, you can use coroutines/asyncio without ever touching a
Future.  Likewise, you can use a Future without ever touching a
coroutine or asyncio.  Note that its `.result()` call isn't a
coroutine.

** They can still encapsulate each other.

A coroutine can encapsulate a Future simply by `await`ing it.

A Future can encapsulate a coroutine with [[https://docs.python.org/3/library/asyncio-task.html#asyncio.ensure_future][asyncio.ensure_future()]] or
the event loop's [[https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.create_task][.create_task()]].

** Futures can implement asychronicity(?) differently

The ability to make a Future from a coroutine was mentioned above;
that's [[https://docs.python.org/3/library/asyncio-task.html#task][asyncio.Task]], an implementation of [[https://docs.python.org/3/library/asyncio-task.html#future][asyncio.Future]], but it's not
the only way to make a Future.

[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future][concurrent.futures.Future]] is another mostly-compatible way.  Its
[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor][ThreadPoolExecutor]] provides Futures based on separate threads, and its
[[https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor][ProcessPoolExecutor]] provides Futures based on separate processes.

** Futures are always paired with some running context.

That is, a Future is already "started" - running, or scheduled to run,
or already ran, or something along those lines, and this is why it has
semantics for things like cancellation.

A coroutine by itself is not.  The closest analogue is [[https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Handle][asyncio.Handle]]
which is available only when a coroutine has been scheduled to run.

* Other Event Loops

[[https://pypi.python.org/pypi/Quamash][Quamash]] implements an asyncio event loop inside of Qt, and I used this
on a project.  I ran into many issues with this combination.  Qt's
juggling of multiple event loops seemed to cause many problems here,
and I still have some unsolved issues in which calls
`run_until_complete` cause coroutines to die early with an exception
because the event loop appears to have died.  This came up regularly
for me because of how often I would want a Qt slot to queue a task in
the background, and it seems this is an acknowledge [[https://github.com/harvimt/quamash/issues/33][issue]].

There is also [[https://github.com/MagicStack/uvloop\][uvloop]].  I have no need for extra performance (nor could
I really use it alongside Qt), but it's helpful to know about.

# Also: What about coroutine generators?
# Are they anything special?