It is commonly understood by practitioners of software that it is a complex endeavour.
We don’t have to be too precise on the definition of “complex”. What I’m interested in here is simply that aspect of the thing where a change in business logic requires a disproportionate change in code.
That phenomenon which leads to the familiar situation of stakeholders wondering why a simple change in business logic requires so much expense in software engineering time and effort.
Why is software complex?
Paradigms
It is telling that the dominant explicit paradigm for writing software is called “object-oriented programming”.
Some might argue that the dominant paradigm is in fact “imperative”, but for my purposes here “imperative” is not so much a paradigm as a lack of one. Ultimately, code is a translation from the language/domain of business logic into the language/domain of computer instructions. Code is hence imperative to the extent there is a lack of such a translation.
So it would seem that explicitly, the dominant paradigm is object-oriented, and implicitly the dominant paradigm is imperative.
Diagnosis
There are two symptoms: object-orientation as a focus on objects instead of relations1; and being imperative as a lack of compilation2. But both of these being symptoms of the same underlying disease, which is an implicit understanding of meaning as a correspondence instead of coherence affair.
For example, we might model users with a User class, along with User properties and User methods. Here the focus is on the data representation of a User, and what associated data a User has, and what things a User can do or can have itself done to.
But what if our users have Money, and they can trade. We have not modelled that at all! Now we can double down and create a Trade or Money class, but does that really help us express invariants like Alice’s account being credited if and only if Bob’s account is debited? In this case, it is not so much the fact that User objects have Money which matters, but the more abstract idea of the relation of Trade as something which simultaneously credits an account and debits another.
Once we shift the focus to the Trade relation, we start seeing things like Alice owing Bob $X and Bob owing Carmen $X being equivalent to Alice owing Carmen $X. Which is the idea of transferable credit/debt3, or the idea of clearing4, considerations of which are ultimately more important when it comes to the expression of business logic in code.
Translation
Writing software requires a translation between business logic and computer instructions.
A lack of compilation means we have to stoop down to the level of computer instructions, resulting in imperative code.
A lack of a focus on relations causes us to write translations “point-wise” for each object, leading to the network of relations at the business-logic level not being cleanly and correctly embedded into the computer instructions level.
Complexity
Imperative code on its own does not lead to software complexity, because being simply a lack of translation, it doesn’t add any complexity beyond what’s already there in the business logic. There may be more “degrees of freedom” at the level of computer instructions, making it harder to “see the forest for the trees”, but there wouldn’t be that familiar and frustrating phenomenon of a change in business logic leading to a disproportionate change in code.
It is only when we do attempt to translate, but do it badly, as in the case of object-oriented programming, where we run into trouble. That we neglect to translate the relations doesn’t mean they are not there, only that they are implicit. Not only that, there are relations at the lower level of computer instructions too! And because the higher level relations weren’t embedded cleanly at the lower level, they get tangled, and this is essentially where multiplicative complexity comes from! Leading again to the familiar and frustrating condition of the programmer simultaneously juggling business logic and computer instruction considerations.
How can software be simple?
Interpreters
Theoretically there seems to be a simple solution: focus on writing interpreters/compilers.
When we do that we are automatically forced to respect relations. For example if we use free monads, our interpreter is forced to be a natural transformation5:
foldFree :: Monad m => (forall x. f x -> m x) -> Free f a -> m a
Then we would automatically have the good things in life like being able to test for business logic without also having to perform side effects like the launching of missiles.
Glimpses of what is possible
In the wild, we do see the benefits of taking compilation or respect for relations seriously.
With Haskell, higher-kinded polymorphism allows for natural transformations, which are an expression of relation-preserving translation between domains. Its compiler is however a monolithic one, even as writing DSLs is a common practice among Haskellers.
Lisp with its homoiconic syntax is good for compilers/interpreters. The syntax being simple allows Lisp code to rewrite Lisp code, making translation relatively straightforward. However, Lisp being untyped and lacking higher-kinded types makes it all to easy to neglect naturality (the preservation/respect of relations) when performing translations.
Why software sucks.
Why not?
Sociologically and economically though, this is difficult, and may very well be unfeasible.
Economically, writing interpreters requires a lot of effort. It takes judgement and experience and often simply a lot of blind thrashing about, to identify the relevant relations, and to keep them in mind when expressing their translation in code. It’s far cheaper to just focus on modelling the business domain pointwise and take care of the relations as and when they present themselves to be considered.
Sociologically speaking, there are path-dependent and network effects. Software engineers these days aren’t already familiar with writing interpreters. University courses have stopped making compiler courses mandatory. Compiler writing itself has become a specialised niche within the larger software engineering profession where compiler knowledge is concentrated. This leads to the writing of the monolithic compilers for the major programming languages we are familiar with today.
No answers.
I have no answers. The problem of software may very well be… too complex?
https://en.wikipedia.org/wiki/Yoneda_lemma
Compilation is specialised interpretation. See: https://en.wikipedia.org/wiki/Partial_evaluation
https://www.goodreads.com/book/show/19811326-money
https://en.wikipedia.org/wiki/Clearing_(finance)
https://en.wikipedia.org/wiki/Natural_transformation
Great post! It echoes some of the points made in John Backus' ACM Turing Award Lecture...from 1977(!!).
Same as it ever was, etc.
Have you read it? Would be interesting to heart your thoughts.
The lecture was called "Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs"
PDF: https://dl.acm.org/doi/pdf/10.1145/359576.359579
In the lecture he outlines a toy, point-free language called FP as a gesture towards what he means: https://en.wikipedia.org/wiki/FP_(programming_language)
His criticism of Lisp-like languages is also that they have too many degrees of freedom, so that every Lisp codebase eventually becomes written in a program-specific idiolect/dialect of Lisp.