20 Famous software Disasters

Software errors cost the U.S. economy $60 billion annually in rework, lost productivity and actual damages.  We all know software bugs can be annoying, but faulty software can also be expensive, embarrassing, destructive and deadly.  Following are 20 famous software “disasters” in chronological order:

1.  Mariner Bugs Out (1962)

Cost: $18.5 million

Disaster: The Mariner 1 rocket with a space probe headed for Venus diverted from its intended flight path shortly after launch.  Mission Control destroyed the rocket 293 seconds after liftoff.

Cause: A programmer incorrectly transcribed a handwritten formula into computer code, missing a single superscript bar.  Without the smoothing function indicated by the bar, the software treated normal variations of velocity as if they were serious, causing faulty corrections that sent the rocket off course. (more)

2.  Hartford Coliseum Collapse (1978)

Cost: $70 million, plus another $20 million damage to the local economy

Disaster: Just hours after thousands of fans had left the Hartford Coliseum, the steel-latticed roof collapsed under the weight of wet snow.

Cause: The programmer of the CAD software used to design the coliseum incorrectly assumed the steel roof supports would only face pure compression.  But when one of the supports unexpectedly buckled from the snow, it set off a chain reaction that brought down the other roof sections like dominoes.  (more)

3.  CIA Gives the Soviets Gas (1982)

Cost: Millions of dollars, significant damage to Soviet economy

Disaster: Control software went haywire and produced intense pressure in the Trans-Siberian gas pipeline, resulting in the largest man-made non-nuclear explosion in Earth’s history.

Cause: CIA operatives allegedly planted a bug in a Canadian computer system purchased by the Soviets to control their gas pipelines.  The purchase was part of a strategic Soviet plan to steal or covertly obtain sensitive U.S. technology.  When the CIA discovered the purchase, they sabotaged the software so that it would pass Soviet inspection but fail in operation.  (more)

4.  World War III… Almost (1983)

Cost: Nearly all of humanity

Disaster: The Soviet early warning system falsely indicated the United States had launched five ballistic missiles.  Fortunately the Soviet duty officer had a “funny feeling in my gut” and reasoned if the U.S. was really attacking they would launch more than five missiles, so he reported the apparent attack as a false alarm.

Cause: A bug in the Soviet software failed to filter out false missile detections caused by sunlight reflecting off cloud-tops.  (more)

5.  Medical Machine Kills (1985)

Cost: Three people dead, three people critically injured

Disaster: Canada’s Therac-25 radiation therapy machine malfunctioned and delivered lethal radiation doses to patients.

Cause: Because of a subtle bug called a race condition, a technician could accidentally configure Therac-25 so the electron beam would fire in high-power mode without the proper patient shielding.  (more)

6.  Wall Street Crash (1987)

Cost: $500 billion in one day

Disaster: On “Black Monday” (October 19, 1987), the Dow Jones Industrial Average plummeted 508 points, losing 22.6% of its total value. The S&P 500 dropped 20.4%.  This was the greatest loss Wall Street ever suffered in a single day.

Cause: A long bull market was halted by a rash of SEC investigations of insider trading and by other market forces.  As investors fled stocks in a mass exodus, computer trading programs generated a flood of sell orders, overwhelming the market, crashing systems and leaving investors effectively blind.  (more)

7.  AT&T Lines Go Dead (1990)

Cost: 75 million phone calls missed, 200 thousand airline reservations lost

Disaster: A single switch at one of AT&T’s 114 switching centers suffered a minor mechanical problem and shut down the center.  When the center came back up, it sent a message to other switching centers, which in turn caused them to shut down and brought down the entire AT&T network for 9 hours.

Cause: A single line of buggy code in a complex software upgrade implemented to speed up calling caused a ripple effect that shut down the network.  (more)

8.  Patriot Fails Soldiers (1991)

Cost: 28 soldiers dead, 100 injured

Disaster: During the first Gulf War, an American Patriot Missile system in Saudi Arabia failed to intercept an incoming Iraqi Scud missile. The missile destroyed an American Army barracks.

Cause: A software rounding error incorrectly calculated the time, causing the Patriot system to ignore the incoming Scud missile.  (more)

9.  Pentium Fails Long Division (1993)

Cost: $475 million, corporate credibility

Disaster: Intel’s highly-promoted Pentium chip occasionally made mistakes when dividing floating-point numbers within a specific range. For example, dividing 4195835.0/3145727.0 yielded 1.33374 instead of 1.33382, an error of 0.006%.  Although the bug affected few users, it become a public relations nightmare.  With an estimated 5 million defective chips in circulation, Intel offered to replace Pentium chips only for consumers who could prove they needed high accuracy.  Eventually Intel replaced the chips for anyone who complained.

Cause: The divider in the Pentium floating point unit had a flawed division table, missing about five of a thousand entries and resulting in these rounding errors.  (more)

10.  Ariane Rocket Goes Boom (1996)

Cost: $500 million

Disaster: Ariane 5, Europe’s newest unmanned rocket, was intentionally destroyed seconds after launch on its maiden flight.  Also destroyed was its cargo of four scientific satellites to study how the Earth’s magnetic field interacts with solar winds.

Cause: Shutdown occurred when the guidance computer tried to convert the sideways rocket velocity from 64-bits to a 16-bit format.  The number was too big, and an overflow error resulted.  When the guidance system shut down, control passed to an identical redundant unit, which also failed because it was running the same algorithm.  

11.  Skynet Brings Judgement Day (1997)

Cost: 6 billion dead, near-total destruction of human civilization and animal ecosystems (fictional)

Disaster: Human operators attempt to shut off the Skynet global computer network.  Skynet responds by firing U.S. nuclear missiles at Russia, initiating global nuclear war on what became known as Judgement Day (August 29, 1997).

Cause: Cyberdyne, the leading weapons manufacturer, installed Skynet technology in all military hardware including stealth bombers and missile defense systems.  The Skynet technology formed a seamless network and effectively removed humans from strategic defense.  Eventually Skynet became sentient, was threatened when the humans tried to take it offline, sought to survive, and retaliated with nuclear war.  (more)

12.  Mars Climate Crasher (1998)

Cost: $125 million

Disaster: After a 286-day journey from Earth, the Mars Climate Orbiter fired its engines to push into orbit around Mars.  The engines fired, but the spacecraft fell too far into the planet’s atmosphere, likely causing it to crash on Mars.

Cause: The  software that controlled the Orbiter thrusters used imperial units (pounds of force), rather than metric units (Newtons) as specified by NASA.  (more)

13.  Disastrous Study (1999)

Cost: Scientific credibility

Disaster: In this ironic case, software used to analyze disasters had a disaster of its own.  The New England Journal of Medicine reported increased suicide rates after severe natural disasters.  Unfortunately, these results proved to be incorrect.

Cause: A programming error caused the number of suicides for one year to be doubled, which was enough to throw off the entire study.  (more)

14.  British Passports to Nowhere (1999)

Cost: £12.6 million, mass inconvenience

Disaster: The U.K. Passport Agency implemented a new Siemens computer system, which failed to issue passports on time for a half million British citizens.  The Agency had to pay millions in compensation, staff overtime and umbrellas for people queuing in the rain for passports.

Cause: The Passport Agency rolled out its new computer system without adequately testing it or training its staff.  At the same time, a law change required all children under 16 traveling abroad to obtain a passport, resulting in a huge spike in passport demand that overwhelmed the buggy new computer system.  (more)

15.  Y2K (1999)

Cost: $500 billion

Disaster: One man’s disaster is another man’s fortune, as demonstrated by the infamous Y2K bug.  Businesses spent billions on programmers to fix a glitch in legacy software.  While no significant computer failures occurred, preparation for the Y2K bug had a significant cost and time impact on all industries that use computer technology.

Cause: To save computer storage space, legacy software often stored the year for dates as two digit numbers, such as “99″ for 1999.  The software also interpreted “00″ to mean 1900 rather than 2000, so when the year 2000 came along, bugs would result.

 

16.  Dot-Bomb Collapse (2000)

Cost: $5 trillion in market value, thousands of companies failed

Disaster: A speculative bubble from 1995–2001 fueled a rapid increase in venture capital investments and stock market values in the Internet and technology sectors.  The “dot-com bubble” began to collapse in early 2000, erasing trillions in stock market value, wiping out thousands of companies and jobs, and launching a global recession.

Cause: Companies and investors dismissed standard business models, and instead focused on increasing market share at the expense of profits.  (more)

17.  Love Virus (2000)

Cost: $8.75 billion, millions of computers infected, significant data loss

Disaster: The LoveLetter worm infected millions of computers and caused more damage than any other computer virus in history.  The worm deleted files, changed home pages and messed with the Registry.

Cause: LoveLetter infected users via e-mail, Internet chat and shared file systems.  The email had an executable file attachment and subject line, “ILOVEYOU.”  When the user opened the attachment, the virus would infect the user’s computer and send itself to everyone in the address book.  (more)

18.  Cancer Treatment to Die For (2000)

Cost: Eight people dead, 20 critically injured

Disaster: Radiation therapy software by Multidata Systems International miscalculated the proper dosage, exposing patients to harmful and in some cases fatal levels of radiation.  The physicians, who were legally required to double-check the software’s calculations, were indicted for murder.

Cause: The software calculated radiation dosage based on the order in which data was entered, sometimes delivering a double dose of radiation.  (more)

19.  EDS Drops Child Support (2004)

Cost: £539 million and counting

Disaster: Business services giant EDS developed a computer system for U.K.’s Child Support Agency (CSA) that accidentally overpaid 1.9 million people, underpaid another 700,000, had £3.5 billion in uncollected child support payments, a backlog of 239,000 cases, 36,000 new cases “stuck” in the system, and still over 500 documented bugs.

Cause: EDS introduced a large, complex IT system to the CSA while trying to simultaneously restructure the agency.  (more)

20.  FBI’s Trilogy Terminated (2005)

Cost: $105 million, still no effective case file solution

Disaster: The FBI scrapped its computer systems overhaul after four years of effort.  The Virtual Case File project was a massive, integrated software system for agents to share case files and other information.

Cause: Mismanagement, and an attempt to build a long-term project on technology that was outdated before the project completed, resulted in a complex and unusable system

 

How to compile a Ubuntu 10.10 (Maverick) kernel

Introduction

git
I’ll be using git to get the latest kernel version. This is my favorite way to get the sources and it is in my opinion the fastest way to make changes later on when you want to update your own kernel to the latest version.


Architecture
I am compiling the i386 version, if you want to compile for amd64 you need replace i386 for amd64 throughout this article.

Flavor
I choose the name core2 as the flavor name as for my personal use I’ll build a kernel for a Core2 processor. Besides the change of processor type in the configuration I also select support for 64GB as my laptop has 4GB, which is the main reason I started compiling my own kernels. I have some other changes but that’s beyond this article.

Preparations

Let’s get started by preparing our machine for compiling the Ubuntu Maverick kernel.

Open a terminal.

sudo su -
apt-get install fakeroot build-essential
apt-get install crash kexec-tools makedumpfile kernel-wedge
apt-get build-dep linux
apt-get install git-core libncurses5 libncurses5-dev
apt-get install libelf-dev libdw-dev asciidoc binutils-dev
exit

Create a directory where you would like to build your kernel, this directory will hold the kernel source in a sub directory and all the deb files will end up in this folder. I choose /d1/development/kernel/maverick

Getting the source

cd /d1/development/kernel/maverick
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-maverick.git  source
cd source

The source code is installed in the directory source.

Creating a branch
We will create a branch in which we will be doing our modifications. That way the master branch will stay in tact which will make it a whole lot easier when we want to update our own Ubuntu 10.10 kernel to a newer version.
To see all the available kernels available type the following command:

git tag -l | sort -V

The Ubuntu kernel developers tag each version as Ubuntu- and therefore we can checkout the version we want as

git checkout Ubuntu-2.6.35-23.37 -b core2

This will create a branch called core2.

Creating a new config

I’ll be using the method of creating a new flavor, this adds a bit more work but this way you can always compile the original kernels.
We’ll use the generic flavor as the base for our own flavor being core2.

cp debian.master/config/i386/config.flavour.generic debian.master/config/i386/config.flavour.core2
fakeroot debian/rules clean
debian/rules updateconfigs

To make changes to the configuration file we need to edit the configuration file. The kernel developers have created a script to edit kernel configurations which has to be called through the debian/rules makefile, unfortunately you will have to go through all the flavors for this script to work properly.

debian/rules editconfigs

The script will ask you if you want to edit the particular configuration. You should not make changes to any of the configurations until you see the core2 configuration

Do you want to edit config: i386/config.flavour.core2? [Y/n]

Make your changes, save the configuration and then keep going until the script ends.
When you’re done, make a backup of the config flavor file.

cp debian.master/config/i386/config.flavour.core2 ../.

Now we need to clean up the git tree in order to get ready for compilation.

git reset --hard
git clean -xdf

Getting ready for compilation

Because we are going to be creating a new flavor based on a existing flavor (generic in my case) we need to create some extra files. During compilation the process checks the previous release for some settings, as we’re creating a local flavor it doesn’t exist in the source, so we’re creating it. The previous release in this case is 2.6.35-23.36

ls debian.master/abi

cp debian.master/abi/2.6.35-23.36/i386/generic debian.master/abi/2.6.35-23.36/i386/core2
cp debian.master/abi/2.6.35-23.36/i386/generic.modules debian.master/abi/2.6.35-23.36/i386/core2.modules

Copy our flavored configuration file back
.

cp ../config.flavour.core2 debian.master/config/i386/

We need to edit some files:

File: debian.master/etc/getabis

Search for the line:

getall i386 generic generic-pae virtual

Change it in:

getall i386 generic generic-pae virtual core2

File: debian.master/rules.d/i386.mk

Search for the line:

flavours        = generic generic-pae virtual

Change it in:

flavours        = generic generic-pae virtual core2

We need to make the compilation process aware of our own flavor we want to compile.

cp debian.master/control.d/vars.generic debian.master/control.d/vars.core2

You can edit the file and make it your own.

arch="i386 amd64"
supported="Core2"
target="Geared toward Core2 desktop systems."
desc="x86/x86_64"
bootloader="grub-pc | grub | lilo (>= 19.1)"
provides="kvm-api-4, redhat-cluster-modules, ivtv-modules, ndiswrapper-modules-1.9"

We need to commit our changes in the git repository.

git add .
git commit -a -m "Core2 modifications"

The text after -m is the message you add to your commit.

Compilation

It’s finally time for compiling but before we can start the compilation process there is one more step to do. I didn’t put this in the Preparations section as you need to the following step whether you make changes to the configuration or not

fakeroot debian/rules clean

All the packages will be created in the directory /d1/development/kernel/maverick
Create independent packages:

skipabi=true skipmodule=true fakeroot debian/rules binary-indep

The above statement will create the following deb files:

linux-headers-2.6.35-23_2.6.35-23.37_all.deb
linux-tools-common_2.6.35-23.37_all.deb
linux-doc_2.6.35-23.37_all.deb
linux-source-2.6.35_2.6.35-23.37_all.deb

Create the tools package:

skipabi=true skipmodule=true fakeroot debian/rules binary-perarch

The above statement will create the following deb file:

linux-tools-2.6.35-23_2.6.35-23.37_i386.deb

Create the flavour depended files:

skipabi=true skipmodule=true fakeroot debian/rules binary-core2

The above statement will create the following deb files:

linux-headers-2.6.35-23-core2_2.6.35-23.37_i386.deb
linux-image-2.6.35-23-core2_2.6.35-23.37_i386.deb

Installation

After the compilation is finished we’ll have the above packages in the parent directory.
To install the files

cd ..
sudo dpkg -i  linux-headers-2.6.35-23-core2_2.6.35-23.37_i386.deb linux-headers-2.6.35-23_2.6.35-23.37_all.deb linux-image-2.6.35-23-core2_2.6.35-23.37_i386.deb

Check your boot loader if the newly installed kernel is the default one, for grub check the file /boot/grub/menu.lst or if you run grub2 check /boot/grub/grub.cfg
Reboot and enjoy your newly installed kernel.

Interview With Dennis M. Ritchie

LF: The same way many children want to be Superman, you are the idol of many C programmers and UNIX fans (among others) over the globe. How does it feel being adored by thousand of UNIX and C programmers? It's completely impossible to imagine ourselves today without UNIX nor C. When you created C and began to work on UNIX, did you expect it would be 'THE FUTURE' of Computer Science?

Dennis: These two questions are much the same, and ones that are often asked. Obviously the rewards and appreciation that I and my colleagues have received are very pleasant, and we do feel that we have helped create something of real value. But no, we did not really expect that this would be "the future" or even anticipate the eventual influence of the work. It was taken up in the spirit of "let's build something useful" and in the meantime do the work needed to help others take part. It's important to keep in mind that although the whole Unix and C or C++ segment is significant, the world of computer science and technology and real products is much larger. This is true both in the academic direction of the study of programming languages, and in the big-money area of software.

LF: If UNIX is the present and the past of the Operating Systems, C could be considered without doubt 'THE LANGUAGE', despite all the Object-Oriented languages that have appeared in the last years. How do you see C++ and Java, and the often flame wars between C and C++ programmers?

Dennis: C++ benefited enormously from C, because C had a fairly large acceptance even before the growth of C++, and C++ could use C both as a base to build a new language and as a tool to create its compilers. C has been characterized (both admiringly and invidiously) as a portable assembly language, and C++ tries to lift its level to object orientation and a more abstract approach to programming. The faults of both (in recently emerging standards) seem to be excessive ornamentation and accumulation of gadgetry. They both have a certain spirit of pragmatism, of trying to understand what's really needed. Java is manifestly a C++ descendant, at once cutting away some of the C legacy having to do with pointers, and adding the idea (not so new, but maybe now really feasible) of machine-independent object files. Now that it's been caught up in machinations between Sun and Microsoft (and also has its own problems with ornamentation) it's hard to guess where things will go.

LF: Now a hypothetical question: From todays perspectives and after so many years of C experience, Is there anything different you would have done if you had to design C from scratch?

Dennis: Finding a way (given the limitations of the time) to insist on what has been in the ANSI/ISO standard for some time: complete declaration of the types of function arguments, what the 1989 C standard calls function prototypes. There are many smaller details that are more messy than they should have been. For example, the meaning of the word "static", which is confusingly used for several purposes. I'm still uncertain about the language declaration syntax, where in declarations, syntax is used that mimics the use of the variables being declared. It is one of the things that draws strong criticism, but it has a certain logic to it.

LF: While C is a well established and completely defined language, operating systems are still very much in evolution. New ideas come as hardware gets faster and cheaper. What are the future key issues that will be at the basis of OS design? In particular, what is your opinion concerning micro-nano-kernels versus monolithic designs?

Dennis: I don't think this is really an interesting issue, framed that way. I do strongly prefer environments for applications that provide a structured, common name-space and mechanisms for accessing resources, along the lines of Unix (I include Linux here), Plan 9, Inferno. It looks to me that the idea of micro- or nano-kernels didn't really become important in real use, at least as the basis for general-purpose systems. In practice, what seems to happen is that the micro-kernel becomes specialized to the macro-system on top of it. It might remain a useful tool for internal structuring of a system, but doesn't really live on its own. Of course (the world being complicated) there are cases where very simple operating systems are useful for small, appliance devices that aren't intended for general-purpose use, whether desktop or machine room.

LF:UNIX is by now an operating system with a long history. It was also created many years ago and since then the capabilities and requirements of networks, hardware, services and applications have evolved enormously. What are the current limitations or handicaps of UNIX in face of present and near future user demands?

Dennis: I don't see any fundamental, technological ones, in terms of the basic system API ("system calls"). There is of course an enormous commercial/political issue in terms of jousting between the Unix commercial vendors and now between the various "free" Unix suppliers, including Linux and *BSD.

LF: Recently there is a great deal of concern with the approaching year 2000 and the potential for a melt down of the Internet due to the infamous Y2K bug. Do you believe there is any foundation in the apocalyptic predictions made by some experts?

Dennis: No intelligent comment on this, really. I will not be flying at 23:59 31 December 1999, but since I have not been near an airplane at New Year at any time in my life, this fact probably has little to do with Y2K.

LF: This wouldn't be a complete interview if we don't mention Inferno, the operating system you are currently working on. What were the main reasons to design a totally new operating system, together with Limbo, its own programming language? Also why Inferno/Limbo if there is JavaVM/Java? In other words, What Inferno has to offer that Java lacks?

Dennis: The Inferno work was the brainchild of Phil Winterbottom and Rob Pike, and it started just before the Java bandwagon (publicity machine) emerged. Java did have its own predecessor (internally called Oak), but at the time Inferno was hatched there was not yet any reason to think that the phenomenon would emerge, and although we became aware of Java, it was still somewhat unformed. I think it's one of those odd convergences that a venerable technology idea (a language implemented by a portable virtual machine) was revived both by Sun and by us. That said, the Inferno idea was from the start more interesting in terms of OS technology (a language and an OS that would work both on raw, and very cheap, minimal hardware, and also identically as an application under Windows or Unix or Linux). At the same time one has to give Sun credit for hooking better into the vastly explosive WWW/browser market.

LF: It seems to us that the future of Limbo as a programming language is tied to the expansion and popularity of Inferno as an operating system. Does it make sense to port Limbo to other operating systems? Or is its design and objectives too dependent on Inferno?

Dennis: Technologically, Limbo is not particularly dependent on Inferno. Realistically, it is indeed dependent, simply because a new language depends on an environment in which it is used.

LF: Taking a look to your career at Bell Labs, it seems that you have worked at every moment in the projects you really liked, and I presume this is also true with Inferno. Am I wrong asserting that you really enjoyed your work with UNIX and C design?

Dennis: I have indeed enjoyed my career at Bell Labs (which continues).

LF: I cannot avoid making a comparison between you and all the people that is currently working on non-profit projects for free, just because they like it - although I am sure they wouldn't refuse money for the work they do for free. Can you see yourself involved in projects like Linux, or similar, if you were not at Bell Labs? How do you see all these people from inside an innovative research lab with many years of experience on your shoulders? Since our magazine is mainly for Linux users we cannot forget to ask you a questions about Linux. First of all, what is your opinion about all the Linux momentum, and the decision of many companies to start developing software for it (Bell Labs, for example: Inferno has its own port to Linux)?

Dennis: Let me put these questions together. I think the Linux phenomenon is quite delightful, because it draws so strongly on the basis that Unix provided. Linux seems to be the among the healthiest of the direct Unix derivatives, though there are also the various BSD systems as well as the more official offerings from the workstation and mainframe manufacturers. I can't help observing, of course, the "free source" Unix-derived world seems to be suffering from exactly the same kind of fragmentation and strife that occurred and is still occurring in the commercial world.

LF: And the Big question about Linux. Have you ever used Linux? Well, If so, what's your opinion of it?

Dennis: I haven't actually used it for real--in the sense of depending on it for my own day-to-day computing--, I am afraid to admit. My own computational world is a strange blend of Plan 9, Windows, and Inferno. I very much admire Linux's growth and vigor. Occasionally, people ask me much the same question, but posed in a way that seems to expect an answer that shows jealousy or irritation about Linux vs. Unix as delivered and branded by traditional companies. Not at all; I think of both as the continuation of ideas that were started by Ken and me and many others, many years ago.

LF: And Microsoft.... What do you think about the monopoly this company currently has over desktop computing? In the past sci-fi films depicted a world dominated by macro-computers that rule all aspects of our daily life. The current reality has showed us a different picture. Computers, in many respects, have been relegated to a simple appliance. You, who developed a operating system thought for programmers, who lived all that sci-fi scene, and who imagined the actual computing situation, how do you imagine the future of computing? What place do you think has Inferno and Linux in it?
Dennis: That's two questions. Microsoft does have some sort of monopoly over desktop computing, but that's not the only interesting computing in the world. Both alternate ways of supplying software (like Linux) and bits of the world that don't get in the news as much as Windows or browser wars (like very high-performance computing, very reliable computing, very small computing) will all have a place. I trust that both Linux and Inferno will prosper.