SCHWERPUNKTTHEMA Policing Science: Genetics, Nanotechnology

The paper opens with the question raised by Grundmann and Stehr, as to whether “knowledge policy” may include “the aim of limiting, directing into certain paths, or forbidding the application and further development of knowledge”. It then explores this theme with reference to contemporary developments in biotechnology and nanotechnology, where the objective of knowledge is to enable us to create and modify at will biological entities (including humans and combined species known as “chimeras”), as well as self-assembling mechanical entities, ab initio through recombinant DNA techniques. I argue that a new category of risks is created by the promised technological applications of these forms of knowledge, called “moral risks”, which threatens the ethical basis of human civilization; these are also “catastrophic risks”, in that their negative and evil aspects are virtually unlimited. The paper asks whether our institutional structures, including international conventions, are robust enough to be able to contain such risks within acceptable limits; or alternatively whether these risks themselves should be regarded as unacceptable, a position which would impel us to seek to forbid individuals and nations from acquiring and disseminating the knowledge upon which those technologies are based.

1 Introduction: "Eppur si muove" ("And yet it moves!") At the conference "The Governance of Knowledge", Essen, Germany, September 5-7, 2001, Reiner Grundmann and Nico Stehr presented the background paper "Policing Knowledge: A New Political Field" which poses "the question of social surveillance and regulation of knowledge".They suggest that "knowledge policy" may include "the aim of limiting, directing into certain paths, or forbidding the application and further development of knowledge" (Stehr,  Grundmann 2003; Stehr 2005).If scientific knowledge is included here, as I assume it is, this proposition will not be well received.One of the great founding faiths of modern society is that of the infinite benefits of the liberation of the natural sciences from the intellectual and institutional shackles of dogma, including religion; its inspirational image is that of Galileo before the Inquisition, forced to recant publicly his belief about earth's movement in space, but unyielding in his mind and certain subjectively of his ultimate vindication. 1 Anyone who seeks to challenge this faith is in for a rough ride.Are there forms of knowledge about nature (including a technological capacity to manipulate nature based on them), now envisioned as practical possibilities in foreseeable futures, of which it may be said that they are too dangerous for humanity to possess?Too dangerous, at least, in the hands of that radically imperfect humanity in and around us, including its all-toodelicate veneer of civilization, which now seems prepared to seek that knowledge?And if so, is it even conceivable that one could argue for their suppression on the grounds that, once realized they will inevitably be deployed, to ends so evil, running unhindered into the future, as to destroy the moral basis of civilization? 2 I at least am not ready to answer these questions -although they are being raised by some in the academic community, especially with reference to biotechnology.An editorial earlier this year in New Scientist, commenting on the inadvertent laboratory creation of a virulent engineered virus which could be used as a weapon in biological warfare (see further discussion below), said: There's also the problem that many biologists choose to ignore biotechnology's threats….John Steinbruner of the University of Maryland, College Park, has suggested setting up bodies to oversee areas of biological research.Such bodies could question or even stop research, or decide if results should be published.As Steinbruner is well aware, his proposal strikes at the heart of scientific openness and freedom.But leaving things as they are is not an option.Biotechnology is beginning to show an evil grin.Unless we wipe that smile from its face, we'll live to regret it. 3re I wish only to ruminate on specific themes with reference to a number of potentially catastrophic risks -risks having a dimension that calls into question the future of humanity itself -related to advances in contemporary scientific knowledge.
I define "catastrophic risk" in this sense as the possibility of harms to humans and other

Seite 32
Technikfolgenabschätzung -Theorie und Praxis Nr. 3, 13.Jg., Dezember 2004 entities that call into question the future viability of existing animal species, including our own.Thus these are not only risks to the present generations of living animal species, but also to future (perhaps all future) generations of presently existing species.One well-known risk of this type is what has been called "nuclear winter", the threat of a pervasive environmental catastrophe that could follow a large-scale exchange of nuclear weapons between the United States and the former Soviet Union (now Russia), under the doctrine of "mutually assured destruction."The hypothesis of environmental catastrophe was based on the expectation that the earth's atmosphere would become loaded with particulate matter, blocking much of the solar radiation reaching the earth's surface, perhaps for a period of years (such an event is thought to have occurred following the impact of massive asteroids colliding with the earth). 4In addition, of course, the huge doses of radiation emitted by these exploding weapons would have profound genetic consequences for plants and animals.

The Lords of Creation
Given the existing stockpiles of nuclear weapons, the risks associated with them still exist, although (in view of the political instability in Russia) it is difficult to know whether the probability now is greater or less than before.
But new catastrophic risks are on the horizon, and these have a fundamentally different character that may require very different institutional responses from us.Their common characteristic, considered as basic and applied science and the technological applications made possible through them, is that they are all based on our latest understanding of biological systems through molecular biology.More specifically, their common scientific basis is the capacity to characterize complete genomes and to manipulate them by means of recombinant DNA techniques (or to create DNA-like mechanical structures).The ultimate goal, already envisioned and set as an objective for research, is a knowledge of genomics so complete that living entities (and life-like mechanical entities) could be constructed, or alternatively deconstructed and then rebuilt and varied, ab initio.According to an article published in Science in 1999, researchers working with a microbial parasite sought to characterize and develop "an organism with a minimal genome, the smallest set of genes that confers survival and reproduction": 5 But since each of the 300 genes found to be essential could have multiple functions (pleiotropism), investigators had no way of finding the degree of redundancy and whittling the genome down further.The next logical step: make a synthetic chromosome of just those genes to build a living cell from the ground up.
Considered in their human implications, I regard these developments as giving rise to a new type of catastrophic risk, which I have called "moral risks". 6Gradations of being (inorganic and organic matter, plants, insects, animals, humans) are and always have been a foundation-stone of humanity's ethical and religious systems.More particularly, "self-consciousness" has been regarded as the essential and distinguishing mark of a human being, uniquely; yet as illustrated in the following section we have, apparently even among some senior scientists, an inclination to experiment with "crossing" these dimensions of existence in an almost casual mood.In my opinion very great evils await us in going down that road. 7

A Short List of "Catastrophic Risks"
1.There are risks from the use of future bioengineered pathogens used as weapons or war or terrorism. 8A recent review in Nature listed the following possibilities: 9 a) Transferring genes for antibiotic resistance (e.g., to anthrax or plague, as Russian scientists have done) or pathogenicity (the toxin in botulinin, which could be transferred to E. coli), or simply mixing various traits of different pathogens, all of which is said to be "child's play" for molecular genetics today.b) Through "directed molecular evolution", especially what is called "DNA shuffling", producing "daughter genes" by shattering genes and then recombining gene fragments in ways that change the natural evolutionary pathways of bacteria.c) Creating "synthetic" pathogens, that is, "artificial" bacteria and viruses, by start-ing with a synthesized "minimal genome" which was capable of self-replication (a kind of empty shell), to which "desired" traits could be added at will.d) Creating hybrids of related viral strains.These possibilities multiply as scientists begin publishing the complete DNA sequences of well-known pathogens: "… [G]enomics efforts in laboratories around the world will deliver the complete sequence of more than 70 major bacterial, fungal, and parasitic pathogens of humans, animals and plants in the next year or two…." 10 Scientists working in these areas point out that actually getting engineered viruses and bacteria to survive in the environment, and to be maximally useful as weapons of war and terrorism, would not be easy to do; moreover, defenses against them can be constructed.What we are faced with the advances in molecular genetics, therefore, is an increase in the risks (possible harms) of novel agents being used in these ways for nefarious purposes.2.

5.
There are related risks from accidental or unintended consequences of genomics research, especially from the genetic engineering of viruses and bacteria, which could result from the escape into the environment of virulent new organisms, irrespective of whether these organisms were intended originally for "beneficent" or "malevolent" purposes.
There was a brief flurry of publicity earlier this year when Australian researchers announced that, in engineering the relatively harmless mousepox virus with a gene for the chemical interleukin 4, in an attempt to create a contraceptive vaccine for mice, they had accidentally made the virus exceptionally toxic: "The virus does not directly threaten humans.But splice the IL-4 gene into a human virus and you could create a potent weapon.Add the gene to a pig virus, say, and you could wreck a nation's food supply". 11There are risks to the "nature" of humans and other animals from intended or unintended consequences of genetic manipulations that either introduce reproducible changes into an existing genome (e.g., human or animal germ-line gene therapy), thus modifying existing species, or create en-tirely new variant species.For illustration here, I will confine myself to the example of "chimeras", that is, combined entities made up of parts of the genome of two or more different species, including of course humans.Some molecular biologists apparently already have done casual experiments inserting human DNA into the eggs of other animals and growing the cell mass for a week or so; and there is much speculation as to what would happen if human and chimpanzee DNA were crossed, since chimps share over 98 % of human genes. 12he DNA of all species now on earth is composed of the same four chemical bases, abbreviated A, T, C, G, arranged into two pairs (A/T, C/G), that make up the "ladders" on the double helix of DNA; different combinations of the base-pairs specify one of 20 amino acids, which combine to form various proteins. 13Some scientists are experimenting with adding more chemicals that would act as new bases, so that, for example, there would be six rather than four bases and perhaps three base-pairs.One of the scientists doing this work is Peter Schultz: "Schultz often says living things have only 20 amino acids because God rested on the seventh day.'If He worked on Sunday,' he said, 'what would we look like?'" 14 The self-comparison between Dr. Schultz and God is interesting, to say the least.There have been widely-publicized discussions of certain unique risks to organic life, stemming from possibilities allegedly inherent in the development of robotics and nanotechnology, especially in a nowinfamous paper by Bill Joy (April 2000), Chief Scientist at Sun Microsystems and creator of the "Java" script.Joy wrote: The 21 st -century technologies -genetics, nanotechnology, and robotics (GNR) -are so powerful that they can spawn whole new classes of accidents and abuses.Most dangerously, for the first time, these accidents and abuses are widely within the reach of individuals or small groups….I think it is no exaggeration to say that we are on the cusp of the further perfection of extreme evil, an evil whose possibility spreads well beyond that which weapons of mass destruction bequeathed to the na-

Seite 34
Technikfolgenabschätzung -Theorie und Praxis Nr. 3, 13.Jg., Dezember 2004 tion-states, on to a surprising and terrible empowerment of extreme individuals. 15e link between nanotechnology and biotechnology is fascinating: Although the former works with intrinsically inert materials, it is seeking to turn them into a perfect analogue of a biological (self-assembling) system.One of the leading Canadian scientists in this field, Dragon Petrovic, has explained the quest as follows: In An immediate consequence of the Faustian bargain in obtaining the great power of nanotechnology is that we run a grave risk -the risk that we might destroy the biosphere on which all life depends.As Drexler explains: Tough omnivorous "bacteria" [created by nanotechnology] could outcompete real bacteria: They could spread like blowing pollen, replicate swiftly, and reduce the biosphere to dust in a matter of days….Among the congnoscenti of nanotechnology, this threat has become known as the "gray goo problem".
The "gray goo problem" attracted so much attention that in England the Royal Society and the Royal Academy of Engineering commissioned a special expert report on it: "Nanoscience and nanotechnologies: Opportunities and Uncertainties" (July 2004).This report contained a special appendix on the "problem", which, it suggested, represented a remote and dubious risk; but it also addressed some unique and quite relevant risks, associated with nanotechnologies, which will be a challenge for government regulatory regimes to come to grips with. 18e important point must be emphasized here, namely, that what has been just described are (hypothetical) catastrophic "downside risks", that is, the potential for very great harms to be done through some future technologies that are already on the drawing-boards.For each of these developments there are both "upside benefits", resulting from future applications of these technologies that could bring substantial benefits to us, as well as the potential for "protective" technological innovations that could mitigate, offset, reduce, or even eliminate at least some of the downside risks.To take the example of the engineering of viruses as bioweapons: As a counter to this threat (and also just to reduce the debilitating effects of viral infections on population health), research is under way in molecular genetics to develop new antiviral drugs that can block the infectious action of any viruses at the cellular level (preventing receptor binding, cell penetration, replication, production of viral proteins, and so on). 19Considered as a totality, however, what these conjoined prospects do is to continually "raise the stakes" in our technological game with nature, whereby the new sets of risks and benefits reflect both, and simultaneously, the potential for an upside of hitherto unattainable benefits and a downside of hitherto unimaginable horrors.As discussed in a later section, this entire prospect increases the challenge to our social institutions to manage our technological prowess so as to realize the benefits and avoid the harms, and likewise increases the risk that we will be unable to do so.

What is different today?
There are undoubtedly other types of catastrophic risks, but those introduced above are sufficient for purposes of discussion!My main point is that these newer risks are fundamentally different in character from the case of nuclear win-Technikfolgenabschätzung -Theorie und Praxis Nr. 3, 13.Jg., Dezember 2004 Seite 35 ter, and the difference has to do with the distribution of knowledge and technological capacity relevant to them (thus requiring a very different institutional response).The technologies giving rise to the nuclear winter risk are controlled by just two nation-states and are maintained (for the most part, and until now) under a thick blanket of military security and secrecy, although the smuggling of nuclear materials out of the former Soviet Union is cause for worry.Both the essential theoretical knowledge, and the engineering capacity needed to turn that into weapons, is confined to a relatively small circle of experts and officials.Not so with the new technologies.The catastrophic risk areas listed above stem from current research programs that are widely distributed around the world; moreover, the strongest drivers of them are private corporations, including the large pharmaceutical multi-nationals, acting with full encouragement, support, and incentives from national governments.Especially where the possible health benefits of genetic manipulations are concerned, the combined public-private interests are overwhelmingly supportive, driving the research ahead at an accelerating pace.Governments especially are enthralled with the economic significance of these new technologies, are competing with each other under innovation agendas to capture major shares of the corporate investments, and are loathe to stop and think about unintended consequences.
All of the characteristics of the knowledge and applications in these areas mean that it is extremely difficult even to think about controlling either the process or the results.For one thing, the knowledge is widely distributed among individual scientists; for another, it is widely distributed among private actors (corporations) which have the option of moving their operations on a regular basis, seeking perhaps the least-regulatory-intensive national base on the globe.(Might we expect H. G. Wells' The Island of Doctor Moreau to be replicated many times? 20) Third, the technologies themselves become increasingly "simplified" and thus easier to hide, if necessary; the genetics technologies, for example, can be carried out in small laboratories almost anywhere.Sergei Popov, the Russian scientist who pioneered germ warfare research using recombinant DNA techniques, observed recently: "The whole technology be-comes more and more available.It becomes easier and easier to create new biological entities, and they could be quite dangerous". 21ourth, oversight is inhibited by the lure of truly extraordinary economic and health benefits promised by the new knowledge and technologies.And fifth, just the astonishing pace of innovation itself today makes the prospect of control and regulation a challenge.
During the past year national governments have been scrambling to respond to just a few of the dimensions of these new risks.Most attention has been focused on human cloning, where a few rogue scientists have challenged authorities in various jurisdictions to "try to stop us", and laws prohibiting this technology are being passed rapidly.But this is a relatively crude technology, albeit one which excites public attention, and one wonders whether authorities will become complacent about their ability to control unacceptable technologies due to their experience with this case.(Meanwhile, there are increasing reports that many genetics scientists are "going underground", in the sense that they have stopped talking publicly about their research in progress for fear that public reactions will be hostile and will result in official steps to halt it.) Among the scientists cited in this paper, two (Bill Joy and Ian Ramshaw) have called for urgent action under the Biological and Toxic  Weapons Convention (1975, hereafter BTWC), to provide explicitly for a global oversight effort over some of the new technologies and their applications described earlier.Unfortunately, and ironically in view of what was to happen only two months later, at a meeting of the parties in Australia in July 2001 the United States unexpectedly blocked the process of completing a protocol under the BTWC that would have made the Convention something other than a statement of good intentions, for in its present form it has no provisions for verification or compliance monitoring.The US government has been pressured by its biotechnology industry sector not to agree to a verification protocol, under which inspections of laboratories and other facilities by international teams of experts would be carried out in all the signatory countries, because industry fears that its intellectual property and commercial secrets could be compromised.At the time of writing other signato-

Seite 36
Technikfolgenabschätzung -Theorie und Praxis Nr. 3, 13.Jg., Dezember 2004 ries were considering whether they should proceed to complete the adoption of the verification protocol without US support. 22nfortunately, we know international negotiation to be at the best of times a tedious and protracted process, and there is reason to believe that in this domain it could be fractious and unsuccessful.This is because all of the technologies described represent frontiers of industrial innovation in which great multinational corporations and the national governments which protect their interests (especially the United States) have significant investments; both corporations and governments would be loathe to see those investments and the immense payoffs expected from them jeopardized by an international control regime.A recent article co-authored by a molecular geneticist and a specialist in the international convention on biological weapons has called for an urgent new effort to strengthen verification under the 1975 Convention and to enlist the biomedical research community in an effort to strengthen deterrence against the uses of bio-engineered organisms for war and terrorism. 23

Conclusion
Now is the time for intensive exploration of the theme of policing science and to ask the following types of questions: 1.

4.
Can we characterize a set of new catastrophic risks, as defined here, related to the leadingedge technologies that are being developed?Do these new risks have an essential character that will make them difficult to control, because the knowledge and the technologies will be so widely diffused?Can these risks be confined to acceptable dimensions by the institutional means now at our disposal, including international conventions on prohibitions?If not, what new tools do we need, and how can we get them?Do professional associations of scientists working in these fields have special responsibilities to assist societies in controlling these risks, and if so, are those responsibilities now being discharged adequately? 24at is at risk in this game, now, is the possibility that the tension between science and society will become both unmanageable for institutions and unbearable for individuals, in other words, that the destructive applications of our operational power finally will overwhelm the rest.This possibility arises out of the striking contrast between the pace of change in social and legal institutions (especially international agreements), on the one hand, and in new scientific and technological breakthroughs in the sciences, especially in genomics, including applications relevant to biowarfare and bioterrorism on the other.In the first-mentioned the pace is painfully slow and progress often remains ineffective even after decades of negotiation, as in the case of the Convention on Biological and Toxic Weapons.The second proceeds at a frenetic and steadily-accelerating pace To reduce the probability that change in the second will overwhelm our social and legal capacity to steer technological development away from the zone of catastrophic risks, it is necessary first to get agreement among influential social actors that this is, as described here, a momentous challenge which contemporary society cannot avoid.The first practical test of our resolve in this regard, I believe, is whether influential scientists can be mobilized in the cause, scientists who will reaffirm the need for new oversight structures, to be erected both within the practice of science itself and also in the relation between science and society.Hegel made a remark, I believe, somewhere in his writings, to the effect that only the hand which inflicts a wound can heal it.The wound here is the rupture with the dominant pre-modern relation of humanity and nature, governed by value-laden categories of being, and its replacement by modern science's purely operational orientation to the totality of the natural world.
I will not speculate here on what a healing of that rupture could mean now, at least, not in any "ontological" sense.But in a practical sense, as a matter of public policy, I think it is clear what is required -namely, that the practitioners of science join others in a program to try to bring our operational powers under the control and direction of social institutions that have universal validity, ones that correspond in sufficient measure with the common aspirations of humanity.It is my contention that today's dominant institutions do not have such validity and that, as a result, everyone on earth is at risk of having these powers become in-struments in an Armageddon waged to the bitter end by contending social, ethnic, national, and religious interests.
What remains to be seen is whether the task as defined here can be widely recognized and grasped as such, while there is still time, and whether our scientific enterprise can be steered towards the shelter of a social compact having universal validity. 25If it turns out that despite our best efforts this cannot be done, there will arise a set of other questions that, for now at least, are too abhorrent for many even to consider.These questions have to do with the possibility that, taking both "normal" human passions and human institutional failings into consideration, there may be forms of knowledge that, as a practical matter, are too dangerous for us to possess, and that our only choice is to renounce and suppress such knowledge or suffer the consequences.In mentioning them we go to the heart of the fateful compact between science and society that has set the course for the development of modern society from the seventeenth century onwards, under the program known as the domination of nature.It is likely that contemporary society is not ready to deal with them, at least, not yet.

1) Galileo
Galilei (1564-1642): http://www.rit.edu/~flwstv/galileo.html 2) There is a practical argument to the effect that, since the development and deployment of such knowledge cannot be thwarted, the most prudent course of action is to superintend its progress closely, so that technological antidotes to the potentially most frightful and destructive applications will be ready before they are needed.I regard this as a strong and possibly definitive counter-position to the one posed here in the series of rhetorical questions.Here it must suffice to say that "universal validity" is not an absolute, in the sense that every person must "buy in," but rather is some common orientation that can attract and hold the support of the most influential and enduring cultural traditions around the world.
Scott Foster, "Man-beast hybrid beyond talking stage," The National Post (Toronto, Canada), 22 August 2001, p. A16."Last October, Greenpeace Germany dug up a patent claim for a human-animal hybrid, … U.S.-based Biotransplant and Australia-based Stem Cell Sciences grew a pig-human embryo to 32 cells before ending its life".13) On DNA see the superb graphics and animation at: http://vector.cshl.org/dnaftb/14) Andrew Pollack, "Not Life as we know it," The National Post (Toronto, Canada), 26 July 2001, p. A15 (reprinted from The New York Times).15) Bill Joy, "Why the future doesn't need us," Wired Magazine (http://www.wired.com/wired/archive/8.04joy_pr.html)16) Allen Abel, "The God of Small Things," Saturday Night Magazine (The National Post, Toronto, Canada), 21 & 28 July 2001, pp.34-37.17) Drexler, Engines of Creation, online in its entirety at: http://www.foresight.org/EOC/18) http://www.royalsoc.ac.uk/templates/search/ websearch.cfm?mainpage=/nanotec/ pressmedianov03.htm19) Haseltine, "Genetic Traps for Viruses"; cf.Miller et al., pp.305-307.20) First published in 1896, this is the story of a rogue scientist who sets up a secret scientific research facility on a remote Pacific island in order to pursue vivisectionist experiments on animals and humans.The entire text is available at: http://www.bartleby.com/1001/0.html21) Quoted in Miller et al., p. 304.22) http://www.brad.ac.uk/acad/sbtwc/See especially G. S. Pearson, M. R. Dando, and N. A. Sims, "The US rejection of the Composite Protocol: A huge mistake based on illogical assessments," and G. S Pearson, "Why Biological Weapons present the Greatest Danger," at: http://www.brad.ac.uk/acad/sbtwc/evaluation/ evalu22.pdf23) Fraser and Dando, op.cit., p. 4. 24) The 1975 Asilomar Conference that established some early ground-rules for DNA research at the initiative of the scientific community itself, had a 25 th -anniversary meeting in 2000.At least according to one report, some senior scientists today are doubtful that the "Asilomar model" will prove to be useful in the future for the oversight of problematic applications of DNA research, particularly because of the enormous pressure of commercial interest that has developed in the meantime.See The Scientist 14[7]: 15, 3 April 2000 (http://www.the-scientist.com/yr2000/apr/russo_p15_000403.html)25) There is not time here to develop this concept adequately.