Why Good People Do Bad Things: The Case of Collective Violence

 

Craig Summers and Eric Markusen

Journal of Systems Software, 1992

 

This article extends the emerging debate and discussion over ethical dimen­sions of computer science from issues such as software piracy, viruses, and unauthorized systems entry to the realm of collective violence. We view collective violence as actions by large numbers of people that contributes to large-scale destruction. Several ways in which computer professionals may contribute to actual or potential violence are briefly discussed. Then, to understand how well-meaning computer professionals can do work of the highest technical quality, but which is routinized and isolated from its social effects, we discuss three types of psychosocial mechanisms: (1) psychological-level aspects of one’s own role; (2) bureaucratic factors routinizing individual involvement, and (3) specific factors in scientific and technological work affecting perceived responsibility. To understand why these mechanisms occur, the importance of perceived short-term economic needs for day-to-day living are considered against values and ethics. A predictive model of temporal and social “traps” is outlined that explains when individuals may contribute to harmful projects regardless of social values and human welfare.

 

Introduction

 

Professions in contemporary society can be characterized by four defining features: they possess specialized knowledge; they are important to society; they enjoy a high degree of autonomy and self-regulation; and they are guided by an ideology of public service. [1] The latter two features involve ethics, defined here as moral guidelines for behavior. Thus, most profes­sions have codes of ethics to which all members in good standing are expected to adhere.

However, simply having codes of ethics does not guarantee ethical be­havior. As society and technology change, new situations emerge which create new ethical dilemmas. Also, is students and practitioners of a profes­sion are not carefully instructed about ethical issues and concerns relevant to their profession, it is unlikely that they will guided by them.

Ethics are every bit as relevant to the profession of computer science as they are to other contemporary professions. There has been widespread and influential dissemination of computer technology in recent years, although this profession is still relatively young. . . . Examination of ethical is­sues that relate to computer professionals’—as embodied in this special issue of The Journal of Systems and Software—is therefore both welcome and necessary. Practices such as illegal duplication of software, insertion of harmful viruses, and unauthorized entry and retrieval of private files all need careful exposure and analysis in terms of ethical principles.

This article, however, examines a rather different ethical dimension that is nonetheless relevant to computer scientists. Rather than focus on ethical issues such as viruses, abuse of passwords, privacy, and copyrights, we are concerned with the possibility that computer professionals may lend their expertise to activities and projects that involve harm to other human beings on a large scale. We are, in short, concerned with the relations among computers, ethics, and collective violence. By “collective violence” we mean large-scale destruction to which many people have contributed.

This article has [four] primary objectives, which are examined in the sections that follow. First, we will briefly address the problem of collective violence during the twentieth century. Second, we hope to persuade readers that they should be concerned with the problem of collective violence. Third and fourth, we will summarize relevant literature from psychology and sociology to explain how and why normal individuals—including professionals—contribute to collective violence.

 

 

Collective Violence during the Twentieth Century

 

Anyone who reads the newspaper or watches the news on television is painfully aware of the prevalence of collective violence throughout the world. In this section, we discuss a number of relationships between pro­fessionals and collective violence.

First, collective violence can occur in a wide variety of forms. Warfare, which can take place between nations or groups of nations (international war) as well as between groups within a nation (civil war), is perhaps the most widely recognized and thoroughly studied form of collective violence. Genocide, a term invented only in 1944, refers to the deliberate destruction of groups of human beings because of their racial, ethnic, religious, or political identity. When governments permit and enforce official dis­crimination and violation of human rights—for example, apartheid in South Africa and torture and “disappearances” in Argentina—large num­bers of people suffer and some lose their lives. Likewise, certain cor­porate practices, such as exploitation of the environment or tolerance of dangerous workplace conditions, can hurt many people. Finally, the nuclear arms race, even though it has been justified as a deterrent, poses the ever-present threat of collective violence on an unimaginable scale.

Second, some scholars have argued that the scale of collective violence is greater during this century than at any other period in history [2]. One analyst of genocidal violence estimates that more than 100,000,000 people have been killed by governments during the twentieth century [3]. Another scholar counted 22 wars underway in 1987—more than in any other single year in human history [4]. Military historians and weapons experts argue that the intensity and lethality of war in the present century greatly exceeds anything in history [5, 61. Projections of the possible results of a nuclear war have estimated that more than one billion people could be killed [7] and the planetary ecosystem catastrophically damaged [8]. The unprecedented levels of collective violence probably do not reflect any increase in aggres­siveness or brutality among human beings, but rather their possession of more effective technologies for killing [9].

A third aspect of professionals and collective violence is that most of th& individuals who contribute to collective violence are psychologically nor­mal and motivated by idealistic concerns. Studies of the Holocaust, for example, have found that the vast majority of Nazi perpetrators were

normal people according to currently accepted definitions by the mental health profession” [10, p. 148]. This finding has been corroborated by numerous other scholars [1].

Finally, professions and professionals make crucial contributions to most~ forms of collective violence. Again using the Holocaust as an illustration, there is strong consensus among scholars that educated professionals played indispensable roles in rationalizing and implementing the exter­mination of the Jews [11]. In his study of German doctors in the Holocaust, Robert Lifton [12] found that these health care professionals made crucial contributions to the killing process, even peering through peepholes in the gas chamber doors to determine when the victims were dead.

288 Computing Professionals and Their Ethical Responsibilities

 

Why Computer Professionals Should Be

Concerned about Collective Violence

 

If psychologically normal professionals could be implicated in violence as repugnant and brutal as the Holocaust, it is also conceivable that other professionals could make equally destructive contributions now, par­ticularly if the effects are less apparent. Therefore, the primary reason that computer professionals should be concerned about collective violence is as potential contributors.

One area of potential abuse of information technology is in intelligence—spying n individual citizens and other computer systems4 In 1988, Canadian newspapers obtained a report by Atomic Energy of Canada on its computerized data base tracking the actions of environmental groups [13]. The report also outlined plans for obtaining unauthorized access to other databases. At around the same time, break-ins occurred at the offices of a Member of Parliament and a number of environmental groups [14—16]:

“Computerized records were taken but valuable computer equipment ig­nored. . . . ‘They took seven entire computer systems and left 25 wires dangling,’ said the network’s director” [17].

There are many questionable uses of computers in this one government-related example. The work done by computer professionals in South Africa has even more direct consequences for human welfare. As this is being written, ordinary people are working conscientiously at keyboards in the’ banking system, the government, universities, and software companies, all~ upholding the Apartheid regime These are ordinary, well-educated people, who go home at night to their families. They are not individually malicious, but are still co-opted into maintaining a society where other human beings are systematically starved, dehumanized, and deprived of education, health care, and other basic human rights. Recent legislative changes may im­prove this situation, but so far the injustice has continued.

Computer technology may also adversely effect human welfare through military weapons use. One of the first computer professionals to recognize this was Norbert Wiener, the developer of cybernetics [1 8—20]. A substan­tial portion of government research (in North America as least) is through military agencies [4, 21, 22]. This involves a broad cross-section of scien­tists and researchers who have little or no control over how their published work is subsequently developed or used.

The greatest threat of computers in the military is in nuclear weapons systems. A war fought with nuclear weapons would constitute a human and environmental disaster. Such a war would not be possible without com­puters and computer professionals. Computer professionals contribute to preparations for nuclear war in at least four ways: 1) computers and the professionals who operate them are essential components of the early warning and command and control systems for nuclear weapons. Malfunctions in these systems may be catastrophic [23, 24], yet in an 18-month period in 1979—1980 alone, the U.S. Senate Armed Services Committee reported

151 “serious” false alarms, and 3,703 others [25]; 2) computer profes­sionals help devise and use computer simulations of nuclear war—so-called “war games” [26]. While computer game simulations are designed to alert officials to the uncertainties and complexities involved in the actual use of nuclear weapons, some analysts have expressed concern that this makes preparations for nuclear war routine [26]; 3) computer professionals may obtain scientific results with eventual applications to nuclear weapons. Scientists conduct basic research without knowing how it will be used; and 4) the most direct way in which computer scientists “up the stakes” for global destruction is in the actual design and development of nuclear weapons and missile guidance systems.

Therefore, computer professionals can do work of the highest technical quality, yet be isolated from the potential human costs. Even those com­puter professionals who have no direct involvement with these or other forms of collective violence should nevertheless be concerned about the problem, since they and their families are potential victims.

 

 

How Destructive Professional Work Is Justified

 

It is disturbing and regrettable to have to consider violent images and atrocities in relation to our everyday, comfortable lives. But perhaps rec­ognizing the problems, and that the corporations and government agencies we work for have vested interests independent of human needs, is the first step in differentiating economic practicalities from values and human wel­fare.

In the preceding section, we showed how apparently legitimate work routines can threaten human welfare in the most inhumane ways. There­fore, it is logical to ask how well-meaning individuals perceive their role in the profession. Psychological and social mechanisms related to this are listed in Table 30.1. This is not necessarily intended to be the definitive taxonomy or to cover every possible example, but it should provide a use­ful summary of processes that may be new to the computer professional. These have been defined from the few existing case studies [27—29], autobiographies [30], ethnographies 131] and related theoretical works [32—35].

We have attempted to list mechanisms which are applicable in many different situations. These have been classified as 1) general psychological processes, 2) processes specific to work in large bureaucracies and organi­zations, and 3) mechanisms that allow scientific and technological work independent of social values.

 

Psychological Mechanisms

 

The mind is capable of playing subtle tricks on us. We do not always take the most rational alternative, or pay equal attention to equally important information. Therefore, we are susceptible to the following psychological mechanisms in many different types of dilemmas.

Dissociation. This involves a separation of different parts of conscious knowledge. The effect is to continue thinking and cognitive functioning by isolating incapacitating feelings and emotional responses [29]. It prevents full awareness of disquieting or unsettling information. Lifton and Markusen [29] state that this may ultimately involve “doubling” of one’s personality, as if separate roles or personalities develop for more and less’ humane behavior, It may be invoked when a role at work begins to contradict one’s personal role {36].2 As an illustration, Del Tredici [37] recorded the following dialogue with the spouse of a nuclear plant worker:

 

‘He was just real happy about being hired at Rocky Rats. We were a young couple, expecting a family, and the benefits were very good. The pay was great— you get what they call “hot pay” for working with radiation, so that’s why he wanted the process operator’s job.

 

Did Don ever talk to you about the fact that he was making bombs?

 

‘He never did go into that’ ([37], pp. 173—174).

 

Several other authors have also described dissociation [1, 31, 38]. A simi­lar procedure is often used in everyday life, e.g., when conscious attention is not used in an activity such as driving, changing gears, or locking a door. We can then devote complete attention to something else, such as an ongoing conversation (although we may later find ourselves wondering whether we actually locked that door).

“Psychic numbing” is a type of dissociation. Lifton [39] documented this in nuclear survivors in Hiroshima. He argues that in the nuclear age, it functions to mask the threat of instant extinction in our daily lives. Ironi­cally, it operates in perpetrators as weI1 as victims, and may allow either one to shut out recognition of brutality.

 

Rationalization. This involves after-the-fact explanations of actions. Festinger developed a theory explaining how a post hoc shift in attitudes results from “cognitive dissonance” [40]. When we become aware that our actions contradict our values, we may rearrange our values after the fact to reduce inconsistency. When we are drawn into taking risks, we may adjust our beliefs about the likelihood of negative outcomes. This style of justi­fication for one’s actions is typified by commonly heard explanations for why a particular project was accepted: “Better I do this than someone else”; “If I don’t do this, someone else will.”

 

 

Bureaucratic Factors

 

Most computer scientists work within bureaucracies, often as specialists on sections of large projects. People who work in large organizations are susceptible to the following ways of separating work and values?

Compartmentalization. A diffusion of responsibility tends to occur naturally with complex technology, since technological work relies on numerous different specialists [35]. Therefore, most individuals have only small parts in the ultimate product, for which they do not feel responsible. (There are also situations in which a compartmentalized product is benign, but could be developed in future for either beneficial or harmful applica­tions.) Lempert [27] reports interviews with four engineering students with summer jobs at Lawrence Livermore (nuclear weapons) Labs: “All four seemed to agree that in only a few months one could not possibly make a large enough contribution to feel one had personally helped to develop new nuclear arms” ([27], p. 63). This type of perception then leads to logic of the following sort: “I only—, I don’t actually use them.” One may fill in the blank with any application: “write viruses,” “assemble the weapons," etc.

Although the division of labor in a large project may contribute to knowl­edge compartmentalization, it may also be the case that the “big picture” is purposely withheld. Diffusion of responsibility is explicit in cases of mili­tary compartmentalization for security reasons [30]. This was true of the thousands of people who moved to the Hanford nuclear reservation for a “top secret” project in the 1940s [31]. Soviet scientist and dissident Andrei Sakharov also noted this in the case of Soviet military research: “I was thankful that I was not told everything, despite my high-level security clearance” ([41], p. 268). However, in military or civilian work, compart­mentalization and diffusion of responsibility lead to situations in which no one seems to actually have responsibility, as illustrated by three examples of work that is heavily reliant on computer technology:

 

It’s not like I’m designing the weapons. The guys who design them are in physics. An engineer at Lawrence Livermore (nuclear weapons) Labs ([27], p. 63).

 

Savannah River is the only facility that is producing weapons-grade plutonium to the defense programs. It is also the sole source of tritium. But we don’t have anything to do here with the actual fabrication of weapons. James Gayer, Public Relations Officer for the U.S. Department of Energy, Savannah River Plant, North Carolina ([37], p. 141).

 

Sandia’s role in the U.S. nuclear weapons program extends from applied research through development of new weapons and evaluation of their reliability through­out their stockpile lifetimes. We do not manufacture or assemble weapons components. . . . Sandia does not produce weapons and components. Sandia National (nuclear weapons) Labs ([42], p. 5).

 

A hierarchical authority structure. In a classic study of obedience; Milgram 1321 told individuals in an experiment to administer electric shocks to people making mistakes on a learning tests He found that indi­viduals would follow orders from a stranger to what they thought were life-threatening extents (see update and social applications in Kelman and Ham­ilton [33]). Although computer professionals in most contemporary jobs do not receive explicit orders (except in the military), there can still be penal­ties for not following procedures and instructions from superiors: these include implicit sanctions such as loss of status, or the possibility of being passed over for promotion [30]. The hierarchical authority structure is usually quite clear in most organizations.

It is sometimes argued that technicians and computer professionals should leave decisions about ethics and values to government leaders. Individual employees are not elected, and not authorized to make autono­mous decisions affecting policy [43, 44]. However, this does not recognize the expertise of those directly involved in a particular project. This logic leads to what Johnson calls the “guns for hire” doctrine [45J. This view suggests that computer professionals should let society regulate what is acceptable through government representatives. Noting that the govern­ment cannot always be trusted to provide objective information, however, Sussman [46] states that our “leaders’ deliberate avoidance of true debate, the contempt they show the public during political campaigning, their use and refinement of propaganda techniques, the attentiveness of so many of them to moneyed interests and not to the people generally, are all major causes of resentment and distrust” ([46], p. 49).

Amoral rationality. This is a preoccupation with procedural and technical aspects of work, while ignoring its moral, human, and social implications. The focus is on how to best do a job, with little attention to broader values and social effects~ Responsibility for the work is perceived to be limited to technical aspects. In the Nazi death camps, amoral rationality allowed health professionals to serve as professional killers. Lifton reports that “an S.S. doctor said to me, ‘Ethics was not a word used in Auschwitz. Doctors and others spoke only about bow to do things most efficiently’” [12, p. 294]. Albert Speer, Minister of Armaments and War in the Third Reich and a primary director of slave labor, directly addressed this in a 1944 note to Hitler: “The task that I am to perform is unpolitical. I have felt very good about my work so long as both I and my work were evaluated purely on the basis of my professional performance” ([38], p. 3; [47]). Wooten refers to this as a system of amoral functionalism, “one essentially devoid of morals and ethics in its decision-making process and one con­cerned only with how things get done and not whether they should get done” ([48], p. 21; emphasis in original). Computer science can be similarly promoted as highly technical, but independent of value considerations.

Once more fundamental social considerations are recognized, it becomes apparent that these questions must be addressed first. As the inventor of the hydrogen bomb in the Soviet Union, Sakharov notes that

 

Our reports, and the conferences where we discussed a strategic thermonuclear strike on a potential enemy, transformed the unthinkable and monstrous into a subject for detailed investigation and calculation. It became a fact of life—still hypothetical, but already seen as something possible. I could not stop thinking about this, and I came to realize that the technical, military, and economic problems are secondary; the fundamental issues are political and ethical" [41], p.268.

 

It will be argued in the final section that this way of thinking is reflected in codes of professional ethics and in educational curricula on science and technology.

 

Facilitating Factors in Science and Technology

 

These are processes encountered in professions based on science and tech­nology. Again, they are distorting mechanisms that separate individual value judgments from the collective effects of work.

Technological curiosity. Regardless of the overall consequences, intel­ligent computer systems can be inherently interesting and can distract the worker from thoughts about the ethical implications of his or her work. Chalk describes a “primitive fascination” [20] with new technology (also see [27]). Since any type of basic research has by its nature no direct~ application, this must be a primary motivation for work on many scientific’ projects. Lifton and Markusen 1291 discuss this general “passion for problem solving” in the work of nuclear physicists’. Hayes [49] argues that work has changed as it has become more technology based; this may be due in part to this curiosity. “What mattered was the product’s capacity to provide more interesting work—a capacity that usually dovetailed with the corpo­rate concern for profitability.” However, “among computer professionals, work was so self-referential, so thoroughly personalized, that it no longer required a public rationale in order to yield meaning” ([49], p. 32).

Distancing effects of technology. By operating as an intermediate pro­cessor in some situations, computers make eventual effects seem more distant. Just as pilots dropping bombs are removed from the human suffer­ing that results, computers can remove the human initiator even more from personal involvement. This can occur in time, with contributions to a project or product to be implemented at a later date. A situation more unique to the computer industry, though, is where the human operator is present at the same point in time, but simply removed from the decision-making process: a preplanned procedure is carried through with automated control. (Not that bureaucracies also serve to distance policy makers from front-line effects, and front-line workers from responsibility for policies.)

 

 

Why Destructive Professional Work Occurs: A Predictive Model

 

Taken together, these mechanisms can result in a situation where many highly-trained people work on projects that ultimately have very large human costs. Use of mechanisms such as these could be reinforced by socialization and professional training [30]. Recruitment, selection, and promotion may all depend on one’s ability to go along with routines un­questioningly. The atmosphere in many settings may not allow open dis­cussion of the effects of a project on society and on human welfare, and may emphasize distinct roles and hierarchies (e.g., with the use of uniforms or titles).

These mechanisms are factors affecting or in response to decisions we make. However, it is not the mechanisms per se that cause contributions to collective violence. For example, although obedience to a higher authority is often cited as a cause of irresponsible individual behavior [32, 33], we make autonomous decisions before following orders. We are not reflex­ively and automatically obedient to any higher authority (although we may decide that it is in our interest to be obedient). As another example, disso­ciation can not fundamentally explain behavior in dilemmas at work. We dissociate as a result of an earlier decision or an event. It is not dissociation that causes computer professionals to work on weapons of mass destruc­tion; rather, they may do so because of practical employment needs, but then dissociate knowledge of destructive effects. To better explain these underlying causes, we will now present a predictive model. It explains why we contribute to large-scale risks that are not in our own society's long-term interests, and therefore why mechanisms such as psychic numbing, rationalization, and obedience are needed.

It seems fundamental to the human condition that although we espouse certain values, individual actions ultimately come down to economic prac­ticalities. For example: “Marie is a mother of two living in a small village in Vichy France in 1941 under Nazi control. Everyone is hustling for a position in the new regime, a pass for curfew, a bit of meat; resistance is not an option . . .“ [50]. The demands of daily living [51] were a priority for survival, and still figure prominently in many cases. But even when ex­treme affluence is attained, the focus on self-interest in the short term does not change. We can see the same process in the following biographical note on a defense electronics executive:

 

RAYTHEON. Thomas L. Phillips, Lexington, Mass. 617-862-6600. SALES:

$8.8 bil. PROFITS: $529 mil. Career path—engineering/technical; tenure—42 years, CEO 22 years. Compensation: 1989 salary & bonus, $1,215,000; owner­ship, 136,000 shares. Not fretting about defense cuts, thanks to his electronics, commercial businesses, now 40% of sales. . . . One soft target: $40 billion Milstar communications satellite—for use after nuclear war. Scheduled to retire at yearend to enjoy New Hampshire lakefront home [52].

 

Of course, wealth is not unethical in and of itself. But certainly when profiting from nuclear war, it is reasonable to wonder how justifications, vested interests, and psychological mechanisms are related. Obviously, day-to-day practicalities for this business executive do not mean actual survival, as they did for the oppressed mother in Nazi-occupied France. In both cases, though, there are immediate, tangible incentives for individuals to contribute to a system in which maximizing their own interests adds to the risk of harm for others later on.

The incentives for decisions that we are faced with can be defined in terms of a number of interacting parameters, such as the value of different alternatives, the probability associated with each alternative, and the type of each alternative [53]. In computer work, one might have to decide between

 

1.  developing a profitable computer project with a 10% chance of even­tual misuse or failure, or

2.  not developing this project, therefore creating no chance of misuse or failure but possibly incurring negative consequences for one’s job.

 

Note that the two alternatives differ in both probability (0% vs. 10%) and value (profit vs. negative consequences). The value can be conceptualized as coming in positive (reinforcing) or negative (punishing) forms. Either type can elicit behavior, although positive incentives are much more desir­able. For example, a programmer would obviously rather work for intellec­tual or monetary rewards, than because he or she was forced to under threat of penalty (e.g., by an oppressive government, or simply because of mone­tary losses).

Parameters such as the value or magnitude of rewards and punishments tend to be relative, rather than absolute. For example, the difference we perceive between $20 and $30 is likely to be seen as more valuable than the difference between $1,020 and $1,030 (also a difference of $10). The interesting thing for dilemmas faced by computer professionals, though, is not a choice based on the perceived value of a single dimension. In alterna­tives where two parameters interact, each parameter has to be weighed, and trade-offs evaluated. Therefore, the computer professional may be faced with choosing between a profitable but low-probability project, for exam­ple, or one which offers less profit but a better chance of success.

Another important parameter in the subjective value of different alterna­tives is time delay. A basic principle of learning theory is that as the delay of a reward increases, its value decreases. Just as the subjective value of an additional $10 varies according to whether it is in the context of $20 or $1,020, $10 received now is likely to be seen as preferable to $10 received tomorrow. This in turn has more value than a promise of $10 or more in five weeks. Interestingly, we can obtain the relative importance of magni­tude and time delay by asking now much money would be equally valu­able: “Would you take $12 tomorrow instead of $10 now?” “Would you take $30 in five weeks instead of $10 now?” Regardless of the actual value in dollars, the psychological value is thus a nonlinear function of time ([54]).

Magnitude and time delay trade off in a predictable manner, although some irrational decisions are produced that do not maximize benefits, as will be discussed below. Rachlin notes the disproportional increase in value of some jobs initially because of this: “In the army . . . you get an enlistment (or reenlistment) bonus so that the delay between signing up and your first pay check is very short” ([53], p. 142). Even advertisements for military service stick to payoffs that are both in one’s self-interest and immediate: “travel . . . summer employment . . . interesting people … earn extra money . . . build on your career . . . part-time adven­ture” [55]. Recruiting has historically appealed to broad patriotic and nationalist values, but these arc apparently not as marketable as early pa~ checks and the promise of more and earlier money, friends, adventure, and jobs. This situation is not unlike that of many computer professionals, for whom a fundamental motivation for many work decisions is economic: the need for a job that satisfies day-to-day needs [51].

A specific model, based on “social traps” [56, 57] relates incentives for individuals in their jobs to larger collective effects. Asia true of all traps, & social trap presents an enticing opportunity, or bait. Like a more tangible trap, a social trap is a situation in which one choice that seems beneficial carries with it other negative consequences. Baron [58] emphasizes that this model is fundamental to dilemmas in many social situations.

 

Because so many situations can be analyzed as social dilemmas, much of the philosophy and psychology of morality is contained in this problem. . . . If everybody lies, we will not be able to depend on each other for information, and we will all lose. Likewise . . . cheating on one’s taxes (making the government spend more money on enforcement), building up arms stocks in the context of an arms race, accepting bribes, polluting the environment, and having too many children are all examples ([58], pp. 399—400).

 

Two different types of traps can be defined, both of which are based on conflicting alternatives. Strictly speaking, “social” traps, or social di­lemmas, apply only to a choice between self-interest and broader social or group interests (e.g., [59]). This model has been formally tested in labora­tory simulations of conflict and cooperation between individuals and be­tween countries [60]. However, there has been practically no attempt to collect empirical data or quantitatively model choices between self- and group interests in real life individual dilemmas, whether political, occupa­tional or ethical.

“Temporal” traps could also be defined, for conflict between an imme­diate, short-term incentive, and a later one. The significance of these choices is that one has to wait to obtain the preferable alternative. Experi­ments with children on delay of gratification have identified cultural and personality variables affecting self-control [61], although the process of weighing different alternatives in decisions is more directly relevant in the present context. Quantitative models have been developed in numerous studies on animal learning defining tradeoffs between parameters such as the magnitude and delay of rewards [62—64]. Nevertheless, until now there have been very few attempts to apply these to the dilemmas that people face.

 

 

For individuals in single-industry towns, the practicality of having to avoid the consequences of unemployment may be much more salient than the possibility of producing a weapon that fuels the arms race [65—67]. Moreover, the weightings that we subjectively give to immediate, local needs over global consequences at some point in the future can be ratio­nalized or overlooked with many of the psychological mechanisms dis­cussed earlier. From interviews with computer professionals, physicists, and engineers working on nuclear weapons, Lempert [27] has noted the motivation that short-term economic needs provides: “in a tight job mar­ket, a young man or woman with a newly-earned degree might abandon a primary academic interest for a tempting salary” ([27], p. 62).

 

It should be clear that some of our decision preferences may be short sighted, and lead us into traps in which there are much larger consequences to suffer. It is also important to emphasize, however, that this model of social-temporal traps does not specify that individuals always choose the short term. Rather, decisions involve weighing the parameters of each alternative and evaluating trade-offs. With other things being equal, the short-term incentive will have greater perceived value.

Looking at decision-making in terms of social and temporal traps is useful for explaining work behavior at all levels of organizational hier­archies. How does the data entry operator perceive and weigh conflicting responsibilities or interests? The model is equally applicable to the execu­tive policymaker.

Although many of the problems of sustainability that we face at the end of the 20th century relate to institutions, organizations, industry, and so on, ultimately these are all made up of individual people. In affirming the importance of individuals and the collective effects of their work, Baron [58] has noted that

 

the problems caused by the existence of social dilemmas are among the most important that human beings have to solve. If we could learn ways to cooperate, wars would disappear and prosperity would prevail . . . more cooperation would solve many other human problems, from conflicts among roommates and family members to problems of protecting the world environment” ([58], pp. 403—404).

 

Practical Applications to Ethical Decision Making

 

The psychological model and collective effects outlined here suggest that the wheels of the technological machine may be powered more by short-term economic interests and psychological, organizational, and technical mechanisms than by actual scientific or social needs (to say nothing of moral and ethical concerns). This can lead to devastating human costs on a world-wide scale. As Bandura [35] notes,

 

Given the many psychological devices for disengagement of moral control, soci­eties cannot rely solely on individuals, however honorable their standards, to provide safeguards against inhumanities. To function humanely, societies must establish effective social safeguards against moral disengagement practices that foster exploitive and destructive conduct ([35], p. 27).

 

In view of this process, then, what practical alternatives are there to facili­tate the choice of the right overall decision, rather than simply the one with immediate rewards?

 

Summary and Conclusions

 

Organizations such as governments, companies, and the military involve many professionals, but have goals independent of human needs. Be­cause of the role computer technology now plays in any large project, computer professionals face ethical decisions between organizational interests and social valu0anfortilnately, if there are vested job interests, the reliance on higher authority, regular routines, and technological curi­osity may support amoral rationality: do a good job technically, but leave responsibility to the larger organization. Because of this process, profes­sionals have been participants in collective violence.

Social and temporal traps provide a useful framework for evaluating the role of individuals in collective violence. These models look at the value and timing (delay) of the alternatives in a decision. Lawful predictions can then be made for both rational and shortsighted behavior. This approach has the advantage of applying to individuals at all levels of organizational hierarchies, and in many different situations.

Finally, in response to the conflicting interests that may arise for com­puter professionals, there are several approaches that may help to structure and prioritize the alternatives. Professional codes of ethics, education, and government policies may all facilitate choices that provide benefits individually and collectively.

 

Notes

 

1.      “Professional” is used here in a broad sense, referring to occupations includ­ing programmers, systems analysts, engineers, technicians, and computer sci­entists.

2.   It should also be recognized that many individuals would not report any conflict between their personal lives and job actions. We are interested in cases, however, where the individual has a vested interest in carrying out organiza­tional goals independent’1’ social values. The psychological mechanisms out­lined show how conflict between vested work interests and values can then be obscured.