Subscribe for updates

Sunday, June 11, 2023

Book Recommendation / Review - The Bias that Divides Us by Keith E. Stanovich

 Book Recommendation / Review - The Bias that Divides Us by Keith E. Stanovich

Introduction

I have been puzzling about the Red / Blue divide in our nation since the 2016 election. This book, The Bias that Divides Us by Keith E. Stanovich has given me some very helpful insights into what is going on in my own brain - and some idea of what might be going on in the “other” side. Given that significant benefit - I would urge everyone I know to read the book. I am writing this to help me better understand and retain what I learned, and to try to persuade you to pick up the book.


That said, the book is a bit of a burden, as 80% of it covers a great deal of research by the author and many others. The author’s command of the language and science is excellent. But I would have profited from a somewhat more succinct explanation of the research foundations.


Since any reader might be similarly challenged, my suggestion is to start with the preface,skip to the last chapter. That  will help you understand the goal of all of the research which is presented up front.  Then go back to chapter 4 and 5. 


To further entice you, these are a few of the insights I gained - I trust you may have some similar ones. If so, please share them. Thanks.


Cognitive elites 

I discovered that I am a “cognitive elite,” and that this is not a positive description. If you are actually reading this, you probably are one too! So pay attention. A cognitive elite has relatively high intelligence, good executive functioning, other positive psychological cognitive skills and usually some advanced education. The problem is that folk like me feel a certain “superiority” to the problems that beset “normal” folk. For example, I think I am resistant to biases that might cloud my judgment, simply because I have studied them, done some personal reflection, etc. The research says that this is generally true. But it turns out that being a member of this “caste” makes me even more likely to be prone to the “myside bias.” 


MySide Bias

“What our society is really suffering from is myside bias: we evaluate evidence, generate evidence, and test hypotheses in a manner biased toward our own prior beliefs, opinions, and attitudes.” (Stanovich, Keith E.. The Bias That Divides Us (p. ix). MIT Press. Kindle Edition.)
Just a note, I will only cite the page number in the rest of this piece.


The important thing here is that this bias comes from beliefs, experiences, opinions, attitudes, values which we hold dear. It is not based on information or ideas or reasoning. There is a sense or feeling of identity, of belonging to a community or tribe, that colors everything we look at. It is not subject to logic or information. In my parlance, this is part of our “gut”, our emotional brain which drives most of our thinking and actions. And the research shows that no level of intelligence or education is protective against this bias. A logical argument will not overcome it.


“But because myside bias is an exception, an outlier, it is the bias where the cognitive elites most often think they are unbiased when in fact they are just as biased as everyone else.” p xi.


In fact,”more cognitively skilled subjects were more likely to have a bias blind spot.” p. 96.


This is hard work because this bias from our history, family, faith, upbringing is very powerful. It gives our lives meaning and purpose. It gives us our community, our sense of identity. It is difficult if not impossible for most people to even consider that those core values are not correct. 


Bear with me a bit, and I will show you some examples that the author pointed out to me.


The Great Divide

Since the 2016 election, we have seen a large split between our ideological political parties. The purpose of this book is to help us understand some of the causes of this division, and work out ways to overcome it. That seems a worthwhile enterprise - so let’s be about it.


Most people are not consistent

This is not news, but it is based on good research. It is a lot of work to align all of your ideas and beliefs, and most of us really do not spend much time doing that. We have a working set, and that is it. I have had the good fortune to spend a bit of time thinking and teaching about some aspects of life issues. That helped me work out a few things - but most people do not get that kind of opportunity. And I know I still have a lot of inconsistencies in my values and judgments.


Recognize That, within Yourself, You Have Conflicting Values 

For one example, the global warming issue. 

“Most conservatives do care about the state of the environment and about the implications of global warming. Most liberals do understand that economic growth reduces poverty as well as hardship. Often both groups do know the facts—they just give different weightings to the value trade-offs.” p. 130

Abortion is another topic where values and beliefs dominate any rational thought or argument. For myself, I have always done my best to not take sides on this one, even as a political candidate. Being ambivalent on this is generally not a popular stance. 


Recognize that “You did not think your way to that”

You are the product of a life of experiences and relationships, not abstract thoughts. Most of our information processing occurs without any awareness on our part. It becomes part of the fabric of who we are - our world view, our basic values. 

Our resident beliefs are the result of interacting with the ideas of others. They are the products of our lifetime of experience. But decades of research in cognitive science have taught us that much of our information processing occurs beyond our awareness. This is particularly true when new information melds well with preexisting biological substrates in our brains. We tend to overestimate how much we have consciously thought our way to our beliefs. Like successful business people who overestimate how much of their wealth was due to their own unique creativity and efforts, most of us tend to overestimate how much conscious thought we have used to arrive at our strongest opinions and convictions. P. 134


Most people are not concerned / aware / tuned in

You may be fascinated by a topic or area of life - like politics or economics - but most of us really are not. A Personal Story: The morning after the 2016 election I went to the local hospital for my normal Wednesday volunteer work in the chemo infusion lab. There were two other volunteers there, and their first topic of conversation was: “Why is the news so upset about the outcome of the election.” I was kind of stunned. They simply did not think it was that big a deal.


Many people do not vote, and do not care about the outcomes. The data is hard to get, but a decent estimate is that only about 2/3 of eligible voters are registered, and about 1/3 of that group does not vote. My math skills are not great, but I think that means that only about 45% of the population actually vote. And of that number, based purely on my personal observation in years as an election judge, only about 25% of them have any detailed information about the candidates and issues. In a primary election, I have had more than one voter return for a new ballot because the voting machine rejected a ballot where every candidate for a specific party was selected. The voter honestly did not understand the problem. Some would ask me to tell them who their party was supporting. 


All News Sources are biased

Recognize that your news source is biased. It may be more truthful (mine tend to not repeat falsehoods :-) but they are NOT evenhanded. Most selectively promote a political agenda because they must cater to their audience. News is a business, not a public service. If they do not alter the facts, at the very least they focus on the stance of their audience. PBS and NPR might be exceptions, but their impact is small. If you want to see a good attempt at a truly balanced news source, I suggest Improve the News.


Some topics are not amenable to rational discussion

The author makes the point that we have made great progress in many areas of human life that are solvable with rational abilities - like poverty and violence. 


But other problems on this list—such as climate change, pollution, terrorism, income inequality, and a divided society—are of an altogether different kind from poverty and violence. For some of these, we may be looking not at problems we would expect to solve with greater intelligence, rationality, or knowledge, but rather at problems that arise from conflicting values in a society with diverse worldviews. P. 128


The case of income inequality provides another illustration of the trade-offs involved in solving some societal problems. Political disputes about income inequality are conflicts between those who have differing values—not between those who are knowledgeable and those who are not. There is no optimal level of income inequality. P. 130


In short, the “problem” of income inequality does not have a unique solution. Disagreements about income inequality arise from differences in values—not because one segment of the population has knowledge that the other lacks. P 130


Politicized Issues 

This myside bias is very strong when issues are politicized. The author’s prime example is climate change. Conservatives are saying that they do not like the way the liberals are using the science of climate change to force greater government control of the economy. They are expressing a value, not a fact. An equivalent one is the HPV vaccine. I will let you look that one up. 


Political Parties focus on fear issues

Just because the major political parties are widely separated, that does not mean that all of their members are supporters of all of their issues. People can be very selective in their allegiance.. 

 Most of the increase in polarization, it is important to note, came from an increase in “negative partisanship”—from subjects increasing their negative feelings toward the opposing party rather than from increasing their positive feelings toward their own party. 

Based on the findings of a wide range of studies, most American voters can’t articulate a principle behind their stance on a particular issue and often don’t know their stance on many issues until they hear the stance supported by their own partisan group. P ?


Most people are not ideologs 

Most people are not as ideologically driven as the political system. The political parties on both sides promote extreme views in order to firm up their membership. They are appealing to our tribal need to belong and our fear of the other. 

“This suggests a model where a relatively unideological population, through identification with parties controlled by cognitive elites, is being led to support positions on issues they would not otherwise have held.” 

“Based on the findings of a wide range of studies, most American voters can’t articulate a principle behind their stance on a particular issue and often don’t know their stance on many issues until they hear the stance supported by their own partisan group.” P ??


Do Not Assume Motives

There is a lot of discussion about how Republicans vote against their own interests when they vote for candidates that do not support programs that would actually benefit them. The assumption is that they should be more focused on their own monetary returns. When someone casts a vote, it generally involves much more than their monetary return on investment. The liberal assumption is that their motive has to be self interest and monetary gain - an insulting opinion to say the least. In fact, people will often vote against their own self interest in the name of a larger value, one accruing to the larger society. 


Values dominate character and fitness for office

We also assume that the character and fitness for office should prevail over values and convictions. People generally do not operate that way. That is not a rational approach to choosing a political candidate. It is really about the party that stands with them, that represents their basic beliefs and values. The details are really not that important. Chapter 6 in the book gives many examples and research that supports this.


I call myself an independent and I am critical of both parties. I have on occasion voted for the independent candidate, and even for a GOP candidate - like Dave Durenberger. But most of the time, I go along with the Democratic offering, even when they are far from perfect. Those of us of the progressive faction often think that the members of the conservative one are bereft of reason for supporting candidates that are extreme or unqualified just to stay in the party line. 


Try this thought experiment the author suggests. The next presidential race has these two candidates: Ted Cruz and Al Sharpton. There is no way Al Sharpton is qualified to be president, but most liberals would feel compelled to vote for him. Their fear of Mr. Cruz would be too great.


Common Humanity vs Common Enemy

The author credits Jonathan Haidt and Greg Lukianoff with creating this distinction. Martin Luther King was focused on our common humanity, where we should all have equal access to all parts of human life. Common enemy politics creates a win / lose scenario, where our side must oppose the enemy in order to hold to our status. People are more motivated by fear than opportunity, especially those with a conservative tendency. In a society with an implicit “caste” structure, according benefits to those on the bottom means taking them from others on the top.


Liberal “blind” spots.

Here are some of the author’s critiques of liberal myside biases.

  • Intelligence is largely inheritable
    (I do not share this one for some reason)  Liberals tend to deny the overwhelming consensus in psychological science that intelligence is moderately inheritable and that there is no strong evidence that intelligence tests are biased against minority groups.   We are “science” deniers. P. 117

  • Women are underpaid.
    Liberals are very reluctant to accept the consensus view that women are not underpaid for the same work when studies are controlled for occupational choice and work history. 

  • Phonics works.
    (Another one I do not share) Liberals tend to deny the strong scientific consensus that phonics-based helps most readers, especially those that are struggling the most.

  • Rent Control is a positive thing.
    It is a fact that rent control causes housing shortages and reduction in the quality of housing. 

  • Charter Schools are bad.
    (Again, not one I hold to.) “The expansion of charter schools is opposed by the Democratic Party and a majority of its white voters (Barnum 2019). However, a bundling conflict arises around this issue position because the party purports to be the advocate of minorities—but in fact a much greater percentage of African Americans and Hispanics than of white Democrats support the expansion of charter schools. Of course, the electoral calculus here is no secret. The Democratic Party wants to placate the teacher unions who are part of its coalition.” P. 145

  • Creationism Taint.
    I tend to discount people who believe in creationism as opposed to the evolution of life. But in that one part of their mental model, they are operating on faith and belief, not reason. From Pew Research: Religious Beliefs and Practices Affect Views on Evolution, Big Bang, but Elsewhere Have a Limited Influence on Americans’ Views. 

  • They are Uninformed.
    The liberal elite tend to think that the other side is totally uninformed. There is no proof of that.  The Pew Research NewsIQ survey of 2015 reported that results are typical across both groups. The 2013 survey had similar results. For economics, the science says that people calling themselves “libertarian” or “very conservative” scored higher on economic knowledge than those calling themselves “liberal” or “progressive.” Of course, the survey results are often skewed by how the questions are posed.

  • Conspiracy Theories
    Liberals tend to think that conspiracy theories are more prevalent on the right. But that is primarily because those are the ones that have been studied and publicized. There are as many conspiracy theories on the left which has been confirmed by formal scientific studies. (See the references in the book, page 119) This is a tough one for me, but here’s some objective data: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9307120/
    “ . . . some studies even suggest that the left is more prone to conspiracy theorizing than the right.“ . . .
    And many conspiracies are kept alive by small and vocal groups on social media.
    In a related note, in 2015, the Pew Research Center conducted a survey of 2,000 adults which concluded about 12 percent of liberals and 10 percent of conservatives believed that childhood vaccines are unsafe. 

  • Selective Filters
    It is relatively easy to search for and selectively find facts that support one’s perspective. A good friend of mine put about 10 of these together for me in defense of the 45th President’s accomplishments in office. I had to check, and they were all true.

  • Misinformation
    This does seem to be rampant on both sides - but the ratio of truth to lies seems to me to be conservative biased.

  • Prejudice.
    I tend to think that more intelligent people are less prejudiced, but the research shows that is not the case. It is true that more intelligent people are less prejudiced against blacks and Hispanics, and illegal immigrants. But they are more prejudiced against fundamentalist Christians, big business, the military and working class people. P. 123

Enough Already

There is a lot more information in the book - please read it. 


What To Do About All This?

The author is an academic, and he levies a great deal of blame and calls for change on our universities. The liberal universities seem to be trying to shut down all debate, to sideline issues discussion in favor of ideology. 


Personally, just reading the book has given me a fresh perspective on the “other side.” But I still find it difficult to actually engage a “different minded” person in discussion on any of these sensitive topics. I know the right way to go about this - to focus on shared values, to ask for assistance in understanding their view, etc. But it just does not overcome my personal fear of conflict. I am sorry about that. 


My Personal Fear Problem

If you have heard of the Braver Angels, it is a national movement to advocate more civility and discussion between our Blue people and our Red people. I have researched this group a bit, and communicated with some of the local leadership. One of the cochairs is an old acquaintance. They have invited me to take an active role, but I have found it difficult to get myself to engage the “other” side, even in this kind of structured and safe environmen, with someone I know and respectt. 


That is probably the biggest lesson I have learned from this book and research. I am very conflict adverse, as I have learned at some great personal cost. I am not sure if I am afraid of the potential conflict, or just think it will be a waste of time. We shall see.

----------------------------------------

References


Drummond, Caitlin and Baruch Fischhoff, Individuals with greater science literacy and education have more polarized beliefs on controversial science topics, August 21, 2017

https://www.pnas.org/doi/10.1073/pnas.1704882114 

I found this study was helpful in this context. It is probably more than you ever want to know about how more education, or one’s sense of having more education, creates a bias. The interesting thing is that the bias seems to be much stronger in areas where faith and belief conflict with the information. It is just further confirmation that those “other” folk are not stupid, not evil - they are just normal humans with strong value and faith commitments that override any other type of information. As the article points out, this tendency has the potential to cause real harm when the belief commitment has real impacts.

For example, individuals who self-identify as political conservatives and endorse free-market capitalism are less likely to believe in climate change and express concern about its impacts (1–9). Individuals who report stronger religious beliefs are less likely to support technologies that they view as interfering with divine creation, such as nanotechnology (10), or to accept the science of evolution, if they view it as contradicting the biblical account of creation (11).

In some cases, individuals can affirm their identity by rejecting the scientific consensus with few practical consequences. The Big Bang, as an example, has little role in everyday life. In other cases, though, actions guided by political or religious identity can be costly. Refusing to immunize children creates risk for them as well as for individuals who are medically unable to be vaccinated, such as the elderly or chronically ill; it has caused recent outbreaks of preventable diseases, such as measles and mumps (12). Rejecting the scientific consensus on evolution may lead parents to advocate for school curricula that omit key scientific concepts, adversely affecting science education and support for scientific research. Societies that ignore climate change contribute to global risks, including food insecurity, political instability, and environmental degradation (13).


Thursday, March 23, 2023

Software Design – history and principles – how to fix our software nightmare

 Software Design – history and principles – how to fix our software nightmare.

I first wrote this in 2018 - and just discovered it. I was writing a bit to one of my grandson’s about his career in architecture, and wanted to give him an example of “patterns.” I know it does not fit with most of the stuff here - but I thought it was worth the time when I wrote it. Worst case - just skip this one. I will not be offended. Thanks.

We have had some software design debacles of late. The Affordable Care Act website craziness, and the MN vehicle registration system are two examples. I worked in software for 30 years, I taught software design and architecture, and was a practitioner. I actually stumbled on the right way to do this stuff, and always assumed that everyone else would eventually get the message. I did not create the solution – a bunch of other people created it – but it amazes me how poorly understood the problem is at this late date.

As is my wont in these pieces, I will explain the problem and the solution as succinctly and briefly as possible. Then I will expand on each point with a few examples and stories, to drive it home.

Brief Outline Summary

“Structured Design” is the key. The alternative is spaghetti design, amorphous design, uncontrolled design – whatever else you want to name it. Structured Design has a very simple set of principles:

  1. Single Entry – Single Exit.
    Every component in a software system must have these attributes. You invoke it by a “Perform” or a BAL – but never by a GOTO or a branch. The software component starts up – does its thing – and then does a RETURN or goes back to where it was invoked. It CANNOT go anywhere else. This seems relatively simple, but it is rarely accomplished in practice. I have stories! In structured software methodologies, these components might be called “methods”, or “subroutines”.  This is the MOST important principle, and would, all by itself, solve 90% of the software quality problems. Sometimes you will see this referred to as “object oriented” design, and implemented via languages like Smalltalk, C+, etc., but it can be done in any language, any technical infrastructure.

  2. Reusable Components.
    Every component is designed to be reused. This is also a brilliantly simple principle, but rarely if ever used. Trust me on this. To enable reuse, each component is published to the larger community, and given a descriptive name, so that people can find it when they need it. This also requires that the interface be as flexible and simple as possible, for which see the next principle. This is more focused on productivity, but it also enhances quality. If we figured out how to do something once without errors, why do it again?

  3. Standard Descriptive Interfaces.
    Every invocation of a component follows standard rules for flexible interfaces. By “standard” I mean STANDARD. This means people from an industry or domain have met, discussed and agreed upon what things are named, and how they are described. If it isn’t a standard, you have to make it one before you can use it. The best example of this today is XML. When you invoke a process, the interface requires that you describe the components you are passing, and the components you are expecting, according to a standard protocol, which requires that you use the descriptive terms as part of the interface. If you know html or xml, this will seem obvious, but others might need an example.
    I invoke a routine to calculate premium. I pass it two fields in this way: <Type of risk>Auto</Type of risk>  <Value of risk>1000</Value of risk>. The values are surrounded by the descriptor – and it indicates when it starts, and when it ends. The subroutine likely would return a single field: <Premium>50</Premium>. You get it? Simple, eh? No length variable, no figuring out a location in a string – THIS FIELD RIGHT HERE is the one. It starts here, and ends there.
    The way things used to be done, and the way idiots continue to do it, is to pass to the subroutine a single address pointing at the location of the beginning of the data, or to actually send the data in a bit stream, consisting purely of the data – nothing else. That seems effective and efficient, but totally dangerous. See below for a fuller understanding of how this works.

  4. Use a Common Message Queue.
    This is not an absolute requirement, but a really useful one. To maintain logical isolation, all subroutine invocation is done through a common message queue. This means that the subroutines must have names and published, standardized interfaces – see above. To invoke a subroutine, you name it, and provide the standard descriptive data it needs. The subroutine processes the data, and then RETURNS the result in the same way. This also implies that a QUEUE can only hand the message to one subroutine, and then MUST return the results to the invoker – no one else – see principle 1 above. This might be called a “Message Broker” if you are looking stuff up here.

  5. Use Software Design Patterns.
    An architect by the name of Christopher Alexander first came up with the idea that physical architecture could be reduced to repeatable “patterns” that worked well. A few pioneers in software picked this up and expanded it into software patterns. This is more focused on productivity than quality, but it helps both. If someone has a genius solution to a difficult problem, and they publish it, then everyone else faced with that problem does not need to reinvent the wheel. This is called a software design pattern. The pattern description should appear in a standard journal or thesaurus, which might be collected into reference works, education classes, etc. Just for a simple example, if I described a design pattern as “publish and subscribe”, what does that mean? One use would be where you have a variable or a data item which changes every January 1. You want everyone that uses that variable to get the change as soon as it appears. How to do that? You should keep a catalog of every piece of software that uses it, and then go change them all at about the same time and hope for the best. You could broadcast it to the whole world using the queue above. Hit and miss. Instead, you establish a publish / subscribe relationship. The owner of the data builds a subscription list – anyone that sends that owner a “subscribe” message is put on the list. If they are no longer interested, they send an “unsubscribe” message. When the data changes, the change is sent to all subscribers via a queued message, instantly. Problem solved. 

  6. Use a Framework.
    This is a little more advanced, and it is not as fundamental a requirement, but the benefits are similarly advanced. A “framework” is a collection of sub-routines or methods, constructed around a common problem, or a common industry. Think about it. How much of your HR system is unique to your company? How much is common to a whole raft of companies? Make the common stuff into a standard framework. Publish it, or sell it, and everyone benefits. If you use the design principles above, you can even upgrade it and change it, pretty much on the fly. If you add new functionality, you can drop the new framework right in, and NONE of the new functions will be invoked, because none of the higher routines even know about them. But they can be used immediately – as soon you make the needed changes. Using a standard descriptive data interface and a message queue delivers that capability.

Why Would Anyone Do This?

Back when I was gainfully employed, I was researching this stuff, and found a noted expert who had published several studies on the topics. I asked him to come and visit with the senior staff of our IT division, and we could change the world. I contacted him, we negotiated a fee, and he agreed to come and talk to the senior management. He had one proviso – he had to meet with the division head, the VP, before he would address the group. Fine by me. We set up the appointment, and I accompanied him to the meeting. After introductions and some general discussion about the field and what he was about, he asked the boss, “What do you think the potential is in your staff to improve the productivity and quality of their software development?” The boss said – 10 to 15%. The discussion continued politely for a bit, and then we made our exit. As we walked out, he said – it’s not worth the time and effort. If the boss’s vision is that narrow, you are not going to get anywhere. He and I both knew that the potential for using the above techniques is not a percentage – but a multiple – 2 to 3 times as productive. You don’t need to believe me – look it up.

Why Isn’t Everyone Doing This?

I think it is like a religion. You can explain things to people until you turn blue. If it is contrary to what they believe, it goes nowhere. The field of economics is exactly like this, and politics is as well. The other side of that, once you do see this, you cannot let it go. It is so obvious, you cannot just forget it. But tread carefully and watch your step. If you try to foment a revolution in one big step, the powers that be will class you as a radical and you will be terminated, removed, etc. I know this from experience. If you aren’t THE boss, go with the flow, move it gently, bide your time.

More Details

In this section, we will expand on the above principles, to explain why they work and what they can do. I have seen enough examples of the wrong way to do things that my experience might persuade you whereas the dry explanation of the right way might not. We shall see.

Single Entry Single Exit

You will see this described in more complex fashion in the literature, but let me give you the way I learned it. I took a software design class in 1970 which really brought it home to me. At the time, I was writing a piece of software which would be used as a utility to make global changes to some software instructions called JCL stored in a procedure library. The boss wanted all of the JCL procedures standardized and simplified, and suggested that I write some software to do it rather than trying to code all of the things by hand. It seemed easy enough, but it was a nightmare. The logic was very complex. So, I took a class in “structured design”.

The instructor started with this example, which I used in every class I ever taught on this topic. In this example you have a file with 50 variables or fields. To simplify things a bit, every field can have only 2 states – they are binary – either one or the other value. If you were to build software the standard way, design around functions and inputs and outputs, in order to completely test this software, you would need to test every possible combination of the 50 fields that might appear in the input file. There is a formula for that: 2 to the 50th power minus 1. It means, there are 2 possible states, and those states must be squared for every possible combination – so 50 times, more or less. I actually calculated that number and proceeded to write it out on the black or white board for all of the class to see. Written in scientific notation: 1.1259E+15. OR: 1,125,899,906,842,620. That is 1 quadrillion – the number above trillion. You get that – it’s a LOT of test cases. Guess what – NO ONE is ever going to run all combinations to test that program. In my prior life, there were many examples of software design that had even more complex situations, and software is daily becoming ever more complex. 

Now how can you design software to handle these 50 fields and be able to thoroughly test it? What is principle number 1? One entry, one exit. You design a separate routine to process each field, or each combination of fields in a hierarchical fashion. In Cobol you do a Perform for any routine you need – a perform goes to the routine, and then returns to the caller. ALL subroutines can only process, invoke another subroutine with a perform, and then end or return. NO OTHER branching is allowed. If you structure it as a hierarchy, at the top level, you say, perform premium payment, perform coverage change, perform whatever. You are not going to have 50 routines for 50 fields at that level, but even if you do – that is only 50. At the lowest level, you could have 50 tests times 2 – one for each field and each of the two variables. Then you have a test for each routine that processes any combination of fields – say we have another 50 or 100. Then another test for every routine that invokes any combination of those, etc.  The result might be several hundred test cases, or even a thousand if the thing is really complex. But if you are certain that all routines stay in the confines of their data and code – the problem is greatly simplified. You can test it completely with a modest number of carefully designed test cases. And if a test fails, you know exactly where it happened.

Trust me on this – very few people in the published literature on programming design really understand this and the power it has. Isolating each logical component and structuring them in a hierarchy gives enormous gains in simplicity.

An Example:
I remember one program in particular called “Main Run.” It was a collection of COBOL programs that processed the Personal Lines insurance for a large insurance company. It ran every night, so it could process the payments received, and keep track of the changes to every policy. It was classic spaghetti code. It would run for hours – and it NEVER ran without encountering a problem. The programming staff were on call every single night to try to fix the thing and get it running again. When I heard about this, I came up with a “genius” solution. IBM had a feature on their 360 generation computers called “checkpoint / restart”. Every so many records or minutes, the system would stop the program, and take a complete dump or checkpoint of the state of the program and store it on disk. Then it would start things up again at that point. That way, in case something died in the system, or the power went out, they could restart the program from one of those checkpoints. To take advantage of this feature, I wrote a little software routine which used the STAE routine to catch the program abend in flight. When this routine got control from the OS, it would take a copy of the data in the buffers – which record was being processed – and write that to a disk file, and then let the abend proceed so there was a dump of the problem. Every time the software started a new record, it passed the data to this subroutine and asked – “Have we been here before?” The routine would hold that data and then write it to disk if it caught an abend. And it would read the records on the disk and compare them. If this one caused an abend the first time, the program was instructed to write out a log record and just skip it this time – thank you very much. You could set how many abends you wanted to process – 10 would usually get them through the night. So, instead of testing the thing to death, you just catch the failures and then avoid them. Worked slick. I know for a fact that people used that routine for over 20 years, because I would get questions about it all the time.

In a perverse approach to this same problem from the other angle, the original space shuttle software development team set a “zero defect” goal with their software. After all, any problem there would endanger the lives of the astronauts. Their approach was literally, to run every single possible test against the software. That works, but it is pricey! 

Just so you know, in the original descent to the moon, astronaut Buzz Aldrin had to deal with five alarms from the lunar module computer that it was its memory was full. He had to restart it three times. Imagine the excitement around that. The “memory full” was a known bug! 

As I noted above, be very careful using the term “zero defect”, lest you be labeled a crazy, irrational idealistic nut, and be shown the door. Rather, talk about defect reduction, etc. Slow but easy, we can get there.

Reusable Components

This makes perfect sense, but almost no one is doing this. Instead of looking around for well designed software that can do some parts of what we want to do, virtually every system design starts from scratch. Let’s say you want a really fancy bathroom. You hire a first class architect – he does a wonderful, unique design, just for you. Now you get a contractor to build it. Let me ask you, does the contractor go fell trees, to cut them into 2 x 4s? Does he dig in the dirt and get clay and build a kiln to make the tiles you need for the wall and the floor? Does he refine copper to make the wire, make steel to create the faucets, build a plastic factory for the tub and shower curtain? Of course not – he finds things off the shelf that will do the job. And, amazingly, these things will all work together, because, even more amazingly, they all share STANDARD DESIGN INTERFACES. You can almost buy any sink or tub, and almost any faucet will fit on it. You can get any light fixture, and it will connect to the standard power line. Talk about standardized design. If you want to get real creative, you can design your own sink, but you better have a standard set of holes for the faucet, or you are not going to make it work. Reusable components go hand in hand with the standard design interfaces. Which is our next topic. 

The other side of this problem is that even if people are building reusable components, they are not publishing them for others. At a minimum, within one software operation, you should be able to document and publish “utility software” – standard, reusable components. Not everyone needs to store and update name and address on every person that is connected to your company. Put them all in one system, give it a clear interface with standard data and your problem is solved. If a customer has more than one policy, or is also an agent or a supplier – they still have one address, one bank, etc.

Standard Descriptive Interfaces

This is another key component of this approach, since it reduces enormously the problem of passing data back and forth between systems. It used to be, if you wanted to send information to a government entity, and then get results back, you had to conform to the standard bit oriented data structure published by that agency. And it might be very different from state to state, agency to agency. But, in this modern era, you can define standard terms for every possible data entity. The problems with bits is that they can change, and you can drop one or two, which totally messes up everything. When we were shuttling data back and forth with agents, every time we added a new system, we had to redo the whole business, because no one actually implemented the standards the way they were designed.

Now, with XML and its derivatives, the entire insurance industry has designed a standard set of terms for all of the data elements that exist in this business domain. And, there are tools which can transform data elements from one domain into another, like from building maintenance to insurance. It’s called XSLT – check it out.

For an example, let’s say that 3M wants to shop ALL of its risks for a new insurance bid. Guess how many data elements 3M would have to provide to the underwriters to give them a reasonable guess on the cost. How many? Remember, they are worldwide, and they have numerous plants and facilities, and numerous different businesses. I think it is safe to say that no one is going to do the bid based on all of that data. But, what if building and infrastructure maintenance had an XML standard for every location, every building, every component that they maintain. What if the insurance industry had a standard for every insurable risk and option – that standard actually exists – I helped create the first version. Now, with a handy little conversion tool – I also helped design one of them – you can define an XSLT transform which would accept all of the 3M data and spit out the insurance data. It is the same data – you are just changing the descriptors required by the different industry. Would that we could adopt universal descriptors for common things, like, say name and address and telephone – maybe someday. Until then, you can actually transform from one standard to another.

Having set up your transform utility, you company is now in the position to be able to quote virtually any business they would like, with almost NO EFFORT. I am on the board of a townhome association. We have 111 homes – they are of 8 different architecture styles. We have a heck of a time getting an insurance agent to bid on them – it is just too much work. I know the insurance standard exists – I would be pleased to hand code the 8 different models and identify them in the 111 risks, if I could find any carrier sophisticated enough to pull this off. When will the future arrive? 

Common Message Queue

This seems pretty simple. You set up, or better, buy, a message queuing facility. Every piece of software in your enterprise is connected to this. Every software routine only gets its input from a message that is delivered via this common queue, and it only puts its results back into that queue. You define the standard transactions and standard descriptive data – and you are done. For example, let’s say you are very forward thinking and have designed a common name and address database, so that all of your systems only have to look in one place to locate the contact information for any entity, any risk, any person that you deal with. If you define the interface correctly, all requests for that data, and all changes to it are conducted via a message with a variable amount of data that has standard descriptors attached. Problem resolved. 

In an actual example, a long time ago, my employer had built just such a common database. They had, however, neglected some of the necessary design features. The database was a single entity, but the support staff decided, probably under some pressure from the other programmers, to adapt their interface to almost every requestor. They published 60 interfaces. I had the staff in a design review session, and we asked them how they could support 60 interfaces. They told us that there were 250 interfaces – they only published 50 of them. Mamma mia! What a nightmare. Some of these were passed via sequential files, so nothing could be done in real time. 

Software Design Patterns

Let’s say you are a mechanical engineer. You want to design a throttle regulator. The problem is that the throttle is an in and out gadget – it increases fuel by pushing down, decreases by pulling up. The engine you want to throttle is a rotating gadget – it spins a flywheel or axel, or something that rotates. How do you translate the rotating speed of one gadget into an in and out or up and down motion in another gadget? You need a bit of genius thinking. That, or go to any standard engineering workbook or catalogue and look it up. It has been done – it is as simple as pie. You have the rotating component spin a control with a weighted pair of arms. The faster it spins, the further out the arms go, which pulls up a slide on the center of the rotating gadget, which is connected to your throttle. Image result for design throttle governor diagram


ALL kinds of engineering problems are designed and available just like that. New ones are created and added all the time. If you are a mechanical engineer or electrical engineer or a carpenter or a plumber, you do not design things from scratch – you go look and see what someone has already contributed to the field. There are literally catalogs of software design patterns. But precious few system software engineers even know they exist. People building a motor vehicle registration system should NOT be designing it from scratch – please!

For a simple example of how not to do this, I used to teach a COBOL programming class. And I was good at it. I would teach the students all of the commands, and give them exercises so that they could use them. Then I would throw a final exam at them – here’s the transaction file, the master file, and what required reports. Figure it out and show me a successful test. The students would literally spend weeks trying to figure out how to apply transactions to a master file. I could have shown them that pattern in about 30 minutes, and we could have spent the rest of the time learning more and different design patterns. But that simple idea never occurred to me, and it does not occur to most system developers to this day. In my day, there were no design patterns – we were making the things up. Now there are hundreds if not thousands of them. Look around a bit.

Frameworks

This is getting out there a bit, but a framework is just collections of the above components that are tailored to an industry or a problem. No one designs a new automobile by figuring out how to caste steel and how to make plastic. That infrastructure already exists. You can specify what you want from the suppliers, and they will deliver it. 

Similarly, in the software world, find a framework that works for your problem, and tailor it. Most of the current literature treats a framework as a very generic set of infrastructure – similar to an operating system, or functions of a coding language. Those are very low level frameworks – just above the hardware level. The genius here is to build or find a framework designed for an industry or domain. This type of framework provides the basic functions needed for every part of that business domain, and allows the end user to combine them in creative ways to solve their unique problem. IBM constructed an insurance framework back in the 80s and 90s, and it is still being advertised. Look at the reference listed below to get a decent pitch on how this all works. IBM was the clear leader in this field way back when, and they seem to be marketing a wide variety of industry specific frameworks. 

Conclusion

Once you see this, you can’t forget it, right? Then why on earth are we still building nightmare software? We don’t design buildings or cars that way – why can’t we teach software design and engineering like we teach mechanical engineering? You can do it. I’m pulling for you!

For an OLD but GOOD set of software design principles, you could do worse than this set: https://sites.google.com/site/carlscheider/carls-papers/design-principles

Bibliography

Anonymous, this discussion reaches the conclusion that it is impossible to fully test a highly complex system. I would agree, if they are not using these basic design principles. See what you think.
https://softwareengineering.stackexchange.com/questions/195571/is-it-possible-to-reach-absolute-zero-bug-state-for-large-scale-software?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa 

Fishman, Charles, on Nasa’s zero defect attempt.
https://www.fastcompany.com/28121/they-write-right-stuff 

Gamma, Eric, et al: Design Patterns, 1995.
https://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612/ref=sr_1_2?ie=UTF8&qid=1525208081&sr=8-2&keywords=software+patterns&dpID=51szD9HC9pL&preST=_SX218_BO1,204,203,200_QL40_&dpSrc=srch 

https://en.wikipedia.org/wiki/Software_design_pattern 

Rouse, Margaret – a definition of structured programming – or modular programming. Introduces the ideas “micro services”, “container technology”, “service oriented architecture”.
https://searchsoftwarequality.techtarget.com/definition/structured-programming-modular-programming 

Saltarello, Andrea and Dino Esposito: Design principles and patterns for “software engineering”. OK, but beware, this is from Microsoft, after all.
https://www.microsoftpressstore.com/articles/article.aspx?p=2233326 

https://www.codingame.com/playgrounds/503/design-patterns/origin-of-design-patterns
Nice explanation of Christopher Alexander’s 253 architecture patterns that worked, and how that led to software patterns. Gamma et al applied the idea to software with their book: Design Patterns: Elements of Reusable Object-Oriented Software

Insurance Software Framework:
ftp://public.dhe.ibm.com/software/industries/frameworks/insurance/2010_04_11_Insurance_Framework__Detailed-V4.pdf
http://www.fasttechnology.com/software/components/ 

Software Frameworks:
https://en.wikipedia.org/wiki/Software_framework
https://techterms.com/definition/framework
https://www.quora.com/What-is-a-framework-in-programming
https://www.infoworld.com/article/2902242/application-development/7-reasons-why-frameworks-are-the-new-programming-languages.html
https://www.ibm.com/us-en/marketplace/hospitality-framework 

Fayed, Mohammad E, and Johnson, Ralph E.: Domain Specific Application Frameworks Experience, 1999. https://www.amazon.com/Domain-Specific-Application-Frameworks-Experience-Industry/dp/0471332801 

Governor for a steam turbine
https://en.wikipedia.org/wiki/Steam_turbine_governing