There’s a really interesting trend in software development that I want to try to sort through.
In the past, when software enters a new business, it had to try to argue for its overall utility. It had to make claims for greater efficiency, equal or greater robustness, equal or greater transparency/control, equal or greater security, etc. Because there was an existing system based on paper/ledgers/phones/trust/etc that it had to compete with on an equal playing field. But now that software is in basically every industry has every organizational system mostly or fully computerized, now business cannot really function if the computers go down.
Whats so paradoxical about this, is that now that these businesses and governments are fully dependent on these incredibly complex webs of software that took years to set up and no one really knows how they work or how to replace them, the actual pressure to ensure their robustness/efficiency/security/etc kind of goes out the window, because there is no real replacement if the system goes down — it sort of just has to be kept alive at all costs. Which means that stability issues and security holes are papered over, more computers are thrown at efficiency issues (i.e. the cloud is so popular largely because it automates this approach), armies of support staff are thrown at information governance issues like redaction, etc.
The next step of the paradox is that there is no way to make anyone accountable for this. Trying to make individuals or small groups accountable is incredibly risky as you may be throwing out the one person or organization who is actually able to keep your system alive at all, and the underlying issue may be. a long-standing problem that they may not have been the primary cause of, even if they have been managing it for awhile.
So instead, the costs of bad code design or software management policy is actually in practice socialized to the entire commerce network of the business, or the entire country in the case of government services.
I.e. when Delta’s servers caught on fire, the whole business screeched to a halt and their customers got stranded. But the people running the servers mostly kept their jobs and mostly just made fairly minor changes to try to keep things afloat. And Delta itself mostly kept its customers because few people thought its competitors were actually doing anything fundamentally different that would improve their statistical failure rate.
Actually the main thing that I see as so alarming about the AI hype in software, is that it is so focused on the potential of AI to cut costs and help expand software coverage to minor niche businesses which currently have poor coverage. But it is actually distracting from the fact that larger, more important software services in larger businesses are actually getting worse every year in substantial and measurable ways. Which everyone does expect to increase with AI adoption, and no one in charge cares because everyone knows the costs will be broadly socialized anyways.
When I’ve talked to more senior software engineers who have been around a few decades, the preferred answer is pretty consistent: