#+TITLE: Modularity & Abstraction (working title) #+AUTHOR: Chris Hodapp #+DATE: April 20, 2017 #+TAGS: technobabble Two central concepts that feature prominently anywhere that computers do are _modularity_ and _abstraction_. This is meant very broadly: it applies to designing software, using software, integrating software, and to a lot of hardware as well. It certainly applies elsewhere too, particularly other fields of engineering, but it appears to be particularly crucial anywhere software is involved. They're generally accepted as desireable, but a bit ill-understood at times. It's common to find people who treat "abstraction" as something that always stands in their way, adds overhead, and confuses things. At the same time, it's common to find people who treat modularity as being present anytime something is broken into pieces. "Being abstract is something profoundly different from being vague. The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." E. W. Dijkstra "Modular design hinges on the simplicity and abstract nature of the interface definition between the modules. A design in which the insides of each module need to know all about each other is not a modular design but an arbitrary partitioning of the bits." (Tim Berners-Lee in [[https://www.w3.org/DesignIssues/Principles.html][Principles of Design]].) "Its is not only necessary to make sure your own system is designed to be made of modular parts. It is also necessary to realize that your own system, no matter how big and wonderful it seems now, should always be designed to be a part of another larger system." (Same) *Abstraction* and *modularity* are tied other inextricably. This is because abstractions draw out the boundaries that modules sit inside, and the interfaces (APIs, communication channels) through which they talk. It need not necessarily be a standardized interface or a well-documented boundary, though that helps. *Decoupling*, *loose coupling*, and *tight coupling* relate to this as well. When two things are decoupled, it often suggests something about at least one of them being a module. Available abstractions vary. The language itself may set boundaries on what abstractions can be created, or practically created (a language like Haskell contains various abstractions done largely within the type system that cannot be expressed in many others; languages like Python, Ruby, or JavaScript might have various abstractions that are meaningful only in the context of dynamic typing). Some languages more readily permit the creation of new abstractions, and this might lead to a broader range of abstractions implemented in libraries. The operating system and standard library provide abstractions that may cross languages (what is a process? what is a thread? what is a library? what is a filesystem?). This extends into electronics (where agreed-upon protocols permit interconnectivity and an ignorance of internals that is required for abstraction) where "modules" take on a tangible form. Abstractions also change over time - both in specifics (consult any list of dead protocols and technologies) and in broader classes (consider how we now have entire abstractions devoted to provisioning virtual resources, and consider how CGI and FastCGI used to be pretty widely-used interfaces). In a practical sense: Where someone "factors out" something that occurs in similar or identical form in multiple places (incidentally, "decouples" also works fine as a term), they're often creating a module (from what was factored out) and some number of abstractions (from the break that created). Consider some examples: - Some configurable functionality in a larger application is extracted out into a system of plugins. The details of the application are abstracted over (as far as the plugin cares), and the details of the plugin are abstracted over (as far as the application cares). The API that the application and plugins use to communicate is the new abstraction now available. The plugins are modules, and the application itself is a module of a different sort. (Witness that sometimes another application will implement the same plugin API.) Given the strong ties between modularity and abstraction, the possible and sensible ways to modularize things then also vary. The modules that make sense may change over time. ("Have modern tools tried to keep embodying the 'Unix philosophy'? Have they extended it, even to other forms of abstraction that weren't previously considered?") One reason behind this is more practical in nature: When something is a module unto itself, presumably it is relying on specific abstractions, and it is possible to move this module to other contexts (anything providing the same abstractions) or to replace it with other modules (anything using the same abstractions). Another reason is more abstract: When something it a module unto itself, the way it is designed and implemented often presents more insight into the fundamentals of the problem it is solving. It contains fewer incidental details, and more essential details. * Other fluff I was around to see what was normal for software made on Windows 3.1, Windows 95, and the like. My take is that most of these pieces of software were sufficiently GUI-oriented that they tried to remove most modularity from the user's perspective. Things like scripting and automation were almost solely as afterthoughts, since most interaction was designed explicitly around the GUI. People know that I love Emacs, but I also do believe many of the complaints on how large it is. On the one hand, it is basically its own operating system and within this it has considerable modularity. On the other hand, there is a perfectly usable operating system underneath, and this makes me wonder why it needs explicit support for [[https://www.gnu.org/software/tramp/][network transparency]]. * Fragments - Abstracting over... - Multiple applications - Multiple users - Multiple CPUs - Multiple hosts - Nix, Guix - [[Notes - Distributed stuff notes (from turtl)]] - [[Notes - Paper, 2016-11-13]] - See notes on functional geekery #75 - Jupyter - Any Plan 9 papers? (Will have to dig deep in the archives) - http://plan9.bell-labs.com/sys/doc/ - Tanenbaum vs. Linus war & microkernels - Conjecture: A module is most useful when available in the most general or most accessible context (e.g. Linux commandline tool vs. a Wordpress plugin or an Emacs package) - the TBL quote on least-power sort of corroborates this, but stands separate in some other ways too. - "most general" might not be right here. - Other conjecture attempt: A module's power is related to how many other modules it can communicate with, without requiring substantial adaptation. An abstraction's power is related to the modularity it accomodates. - Another conjecture attempt: An abstraction's power isn't related to how broad it is, but to how well it connects things. A needlessly simplistic abstraction requires a lot of other adaptation to be useful. A needlessly specific one excludes a lot of potential modues. - hypothes.is is a sort of module unto itself here too, trying to remove commenting and annotation from existing, very siloed solutions. - "Self" paper & structural reification? - I'm still not sure how this relates, but it may perhaps relate to how *not* to make things modular (structural reification is a sort of check on the scope of objects/classes) - What by Rich Hickey? - SICP? - https://mitpress.mit.edu/sicp/full-text/sicp/book/node50.html - "On Understanding Data Abstraction, Revisited" - Frameworks Don't Compose ([composition][]) - "On the Criteria to be Used in Decomposing System into Modules" (Barnas) - suckless, and their tools & methodology - https://suckless.org/philosophy - even though they can take things waaaay too far... - Containers? - http://www.catb.org/~esr/writings/taoup/html/apb.html#Baldwin-Clark - Carliss Baldwin and Kim Clark. Design Rules, Vol 1: The Power of Modularity. 2000. MIT Press. ISBN 0-262-024667. - https://colah.github.io/posts/2015-09-NN-Types-FP/ - Was this the one that talked about 'modularity' in deep learning? - NodeRED might be interesting here, but first I need a clear idea of what it factored out into a separate component. - Lua is notable here for the effort spent in making it easy to both embed (e.g. as a scripting or extension language) and extend (e.g. with other C libraries). Guile may be similar. - NeoVim is also an interesting case here as it is designed to be embedded, though I'm not sure what this means yet. - Find the link to Les Hatton's slides (cyclomatic complexity?) on the empirical effects of too many / too large modules - Examples of more 'modern' tools: - socat - jq (the JSON processor) - Nix and some related tools (which take related functionality that is present in numerous PL-specific package managers) - Jupyter * Link-pile: - [[http://www.catb.org/~esr/writings/taoup/html/][The Art of Unix Programming (Eric S. Raymond)]] - [[https://circleci.com/blog/its-the-future/][It's the Future]] - [[https://circleci.com/blog/it-really-is-the-future/][It really is the future]] - [[https://www.youtube.com/watch?v%253Dtc4ROCJYbm0][AT&T Archives: The UNIX Operating System]] - [[http://blog.ezyang.com/2014/08/the-fundamental-problem-of-programming-language-package-management/][The fundamental problem of programming language package management]] - https://www.reddit.com/r/programming/comments/4bjss2/an_11_line_npm_package_called_leftpad_with_only/ - https://www.functionalgeekery.com/episode-75-eric-b-merritt/ - https://www.w3.org/DesignIssues/ - https://www.w3.org/DesignIssues/Modularity.html - http://www.w3.org/DesignIssues/Principles.html - http://www.freecode.com/articles/editorial-the-two-edged-sword - https://clojurefun.wordpress.com/2012/08/17/composition-over-convention/