Scaling software development
Modern software development (be it classical or agile) lacks of scaling. Several approaches like Scrum of scrum, SAFe, LeSS, etc. tried to overcome this issue — and I would argue they failed.
However, scaling large systems is not a unique challenge for software engineering. And even within the computer science community, other disciplines have solved the issues we are still struggling with.
So let’s start with super computers. Super computers are super cool. Software development (not super cool) will follow later.
Concurrent systems
Concurrent systems are ubiquitous. They appear in form of multi-core processors or as large scale cluster computers resulting in supercomputers. Luminary like Edsger Dijkstra, Per Brinch Hansen, and C.A.R. Hoare accomplished pioneering work on concurrent computing.
There are two classes of communication for concurrent systems: shared-memory and message-passing system.
Shared-memory
Shared-memory is the use of a shared memory for communication between threads, cores, or compute units. The advantages of using shared-memory communication are
- User friendly: The global address space provides an easy interface to the memory.
- Fast and uniform: Information sharing between processors is both fast and uniform.
However, this comes with some costs. The disadvantages are
- Scalability: Adding more processors geometrically increases traffic on the memory-processor path
- Implicitness: Communication is implicit and, therefore, hard to optimize.
- Responsibility: The programmer is responsible for synchronizing constructs that ensure correct access of global memory.
Message-passing
Instead of a shared memory, one can also pass messages in networks to communicate between, threads, cores, or compute units. Message-passing communication has the following advantages:
- Scalability: Memory is scalable with number of processors. Each processor can rapidly access its own memory without interference and overhead.
- Responsibility: Elimination of the need for synchronization constructs such as semaphores, monitors, …
- Distributed: Naturally supports distributed computation
But again, no free lunch. The disadvantages are:
- Copy overhead: Information exchange among processors requires effort.
- User unfriendly: No simple mapping of existing data structures (based on global memory)
Transformation to software development
I know you are grateful for this introduction to concurrent computer systems and reminded you to the course on this topic you already took back in the days of studying.
But: what happens if we make the following substitutions:
- processor → developer
- memory/data → work
- programmer/user → management
- computing → development
- semaphores, monitors, … → meetings, standups, …
We get the following description of software development communication types (substitutions marked by italic and now I wished Medium would support more formatting options):
Shared-work
Advantages of shared-work communication are
- management friendly: The global address space provides an easy interface to the work.
- fast and uniform: Work sharing between developers is both fast and uniform.
However, this comes with some costs. The disadvantages are
- Scalability: Adding more developers geometrically increases traffic on the work-developer path.
- Implicitness: Communication is implicit and, therefore, hard to optimize.
- Responsibility: The management is responsible for synchronizing constructs that ensure correct access of global work.
Message passing
Advantages of message passing are
- Scalability: Work is scalable with number of developers. Each developer can rapidly access its own work without interference and without overhead.
- Responsibility: elimination of the need for synchronization constructs such as meetings, standups, …
- Distributed: Naturally supports distributed software development
Disadvantages of message passing are
- Copy overhead: Information exchange among developers requires efforts
- Management unfriendly: No simple mapping of existing work constructs (based on global work)
Conclusion
I am still surprised how well this transformation works.
We all know instances of both development models:
- Shared-work software development: Scrum, Waterfall, V-Model, Kanban, XP, …
- Message-passing software development: Open source and inner source software development
You may agree and wonder what is it good for?
We can transfer from the field of concurrent computing
- theory and mathematical models
- metrics
- optimizations, and
- evaluations.
So if we drive the analogical conclusion further, we see that for smaller, local software development, shared-work (traditional/agile) is more suitable.
However only message-passing software development enables concurrent, large-scale software development — you won’t find a super computer with shared-memory communication.
For message-passing software development
- local work is cheap, information exchange is expensive.
- managers are not important any more — and this scares them.
- Research on scaling software development is research about scaling networks.
And there are some bonus benefits:
Because we human developers can process only communication with a lower information density (e.g. mails) in contrast to large computing systems, we can store store and archive all information exchange over the interconnection network.
All tools we need is already available: internet, git, GitHub (with pull requests), mailing lists, forums, bug tracker, code review tools such as Gerrit, etc.
And the best thing: message-passing software development works — in form of open source and inner software development.
