Comments on: I told you so … open source is better.

By: Anonymous

Anonymous — Mon, 31 Dec 2007 13:57:50 +0000

Mathematicians in my experience do care about this. I've never seen a peer-reviewed paper in a major journal which uses Mathematica or Maple outside of the WZ algorithm... Admittedly, my areas of interest don't often intersect with Mathematica's or Maple's main capabilities.This must be an issue of research areas. Most of my papers, and many of the papers I've read, have user some computer algebra, but the referees have never even mentioned it as an issue (let alone complained about it).I think one issue is that it is rare for any computer algebra code to rely on unique features of a certain system. Usually it is pretty easy to translate it to any other system, and that tends to reassure people (since they know that if they suspected a problem, they could always run it on their favorite system to check, even though in practice they don't do this).outside of the WZ algorithm. (This algorithm generates hand-verifiable certificates of correctness.)In many cases they are indeed hand-verifiable, but I'm pretty sure that most WZ certificates appearing in published proofs have never been checked by hand. This is another thing that is psychologically reassuring (and therefore frequently gets mentioned) but rarely gets taken up. Whenever a calculation is short enough to be doable by hand in principle, the author is likely to highlight this, but here's one piece of evidence: I've seen many papers in which the authors claim something is short enough that it could be checked by hand, but I don't recall any in which the authors announce that they did check it by hand.I do know that Haken and Appel's proof of the 4-color theorem is still a bit unsettling to me despite the fact I understand how it works, the code is completely open source, and the result has been replicated by several researchers in several different ways.I agree that the Appel and Haken proof was unsettling, but not primarily because of the computer use. Rather, it wasn't computerised enough, in the sense that elaborate hand calculations and case analysis were needed to verify that the computer part was actually complete and matched up with the human part. This interface between humand and computer proofs is by far the least reliable part. Humans are good at building conceptual proofs, and computers are great at systematically checking something simple, but this interface combined the weaknesses of both. The only record of it consisted of a lengthy sequence of hand-drawn diagrams published with the 1976 proof, and I do not believe anyone except Appel and Haken ever checked it carefully. For many years, there were rumors of gaps. This is a very good reason to feel unsettled.The only other proof I am aware of is by Robertson, Sanders, Seymour, and Thomas from 1995. It massively simplifies the Appel-Haken proof and in particular cleans up the interface and systematizes the proof to the extent that it is overwhelmingly easier to check. (In fact, they found this proof after trying to check the Appel-Haken proof and giving up.) I am confident that this proof is understandable and reliable.This is a good example of why the overall strategy for a huge computer-assisted proof has to be chosen very carefully and sensibly.However, there are almost no computer-assisted proofs of this scope. There are a handful of others (such as the Kepler conjecture or the projective plane of order 10), but most proofs that use computers are much less complicated and need very little computer time or custom code.I still feel like the 4-color theorem is just a little suspect because no one has ever really understood *why* it is true.I wouldn't call it "suspect" but of course I agree with you. There are two issues here: what constitutes a proof, and what constitutes a beautiful, illuminating proof. Computer-assisted proofs may have some beautiful aspects, but they are almost never as illuminating as completely conceptual proofs. On the other hand, they are frequently every bit as reliable as human proofs.

By: greg laden

greg laden — Mon, 31 Dec 2007 11:22:38 +0000

Lucas,That is very interesting, and it is interesting to learn how proprietary software fits into actual research in math.I don’t think one has to necessarily personally understand or analyze under the hood. For instance, I get the basics of how stable isotopes are measured in samples, I’ve seen it done, read the literature, but I can’t personally tell you if a certain lab is doing right or the details of how they are doing it. But I know a couple of people who absolutely can do this, and I can ask them.

By: Lucas

Lucas — Mon, 31 Dec 2007 00:01:33 +0000

I think that most mathematicians don’t have the time or inclination to look into how even open source software packages work. There is certainly no hope at all that mathematicians (as a group) would understand in intricate detail the often closed-source internal workings of their computers. Do you understand how the CPU in your Linux box works in intricate enough detail to be certain of its workings? On the other hand, when the source is open, the scientists with the inclination will frequently check its correctness (if the software is frequently used in research).Mathematicians in my experience do care about this. I’ve never seen a peer-reviewed paper in a major journal which uses Mathematica or Maple outside of the WZ algorithm. (This algorithm generates hand-verifiable certificates of correctness.) Admittedly, my areas of interest don’t often intersect with Mathematica’s or Maple’s main capabilities. I do know that Haken and Appel’s proof of the 4-color theorem is still a bit unsettling to me despite the fact I understand how it works, the code is completely open source, and the result has been replicated by several researchers in several different ways. I still feel like the 4-color theorem is just a little suspect because no one has ever really understood *why* it is true.I reiterate that using closed-source programs is fine for research, just so long as they don’t appear in the finished product. The situation is much the same as using heuristics, which every mathematician uses constantly. Heuristics likewise don’t appear in peer-reviewed papers.P.S. I didn’t mean to be rude, and apologize if my remarks came off as rude.

By: greg laden

greg laden — Sun, 30 Dec 2007 20:56:33 +0000

This is developing into a very interesting discussion. Funny, all the mathematicians I know personally are much better behaved and more polite.I certainly do not want to defend the idea that this is a bug. I have no idea. But I do think that the arguments that you can have a black box doing your calculations just like lab scientists have black boxes doing their electrophoresis (or whatever) are valid. In fact, they are absurd. It is not true that this approach would be considered acceptable. Increasingly, lab work is outsourced or done with machines that are essentially black box like. I’m particularly familiar with work done along these lines in endocrinology and stable isotope research.The researchers that send samples out without being able to do the work themselves, or who treat their equipment like a black box and do not understand (and are not able to examine, adjust, etc.) what goes on under the hood are the bad ones … they are perhaps even crossing an ethical line.I am shocked to find that some mathematicians do not worry about this issue. Shocked I say!Perhaps many of you are too young to remember, or perhaps simply not well educated in the relevant areas, the MacClade fiasco. Do you? If not, go find out about it. MacClade was a piece of black box software that for quite some time produce phylogenetic analyses (cladograms) that conformed in all ways to expectations. there was no way to tell if the cladograms were correct or as advertized. You could not infer the inner workings by looking at the results. When it was discovered that there was a flaw in the software, and that many of the perfectly normal looking results over many months (a couple of years maybe?) of time and across numerous peer reviewed papers were simply wrong, shit, really bad shit, hit the fan. Some people in tenure-track jobs had their careers trashed, people with tenure were better protected.Perhaps mathematicians have special powers that allow then to know without looking what is going on inside the black box. But I doubt it!

By: Lucas

Lucas — Sun, 30 Dec 2007 15:25:25 +0000

Rob: I agree with that quotation completely. For this reason, I think that if someone were to release a proof which used proprietary software X, then it would be looked upon with even more skepticism than even proofs using computers normally are. However, I doubt that Mathematica or Maple are so clever that it would come to this–indeed for the computer assisted proofs like the 4-color theorem or Kepler’s conjecture, the main problem is size of the computation. For the 4-color theorem, the computations are not particularly hard to do by hand, there are just a *lot* of them. I don’t think that Mathematica exceeds the cleverness of mathematicians, who, if the situation calls for it could reproduce the closed source cleverness of Mathematica. (On the other hand, it once took me 5 hours to evaluate an integral by hand that Mathematica did in less than 1 second.) The philosophical point is still valid, though Laden picked a very poor example to try and make it.”Applying this argument, no scientist should have any equipment or supplies in the laboratory that has not been made by their own fair hands!”But mathematics is different from other sciences. We have something called proof, and we require to be absolute. Consider something like the Riemann hypothesis, which has been verified for an enormous number of potential instances. If it were a physical question, it would be much better supported than general relativity, but mathematicians are agnostic on its truth. I’ve seen one number theorist say in all seriousness that he has no idea if it’s true or not, and he had studied the issue quite a lot.

By: Anonymous

Anonymous — Sun, 30 Dec 2007 14:57:59 +0000

Maybe instead of attacking Dr. Laden you should read the article he is referring toOh, I have read it (I’m a mathematician and read it when it first appeared in Notices of the AMS). I don’t agree with it, and I think Stein and Joyner have an axe to grind, considering that they are both heavily involved in the development of SAGE (Stein started SAGE and Joyner is an active developer). Basically, I think Stein and Joyner have a valid philosophical point, which is that it would be nice if we could inspect or modify anything we wanted. If Wolfram decided he was so rich that he was going to release Mathematica as open source and continue to pay people to work on it, we’d all be better off. However, the underlying philosophical point is fairly minor and is simply not important for the vast majority of mathematical and scientific work. It’s foolish to sacrifice tremendous practical benefits for a tiny theoretical gain. Of course, Stein and Joyner want us to: they are among the tiny group of experts who would benefit the most from open access to the source code, plus they benefit greatly from having many people use SAGE.Plus I’m not attacking Dr. Laden, who seems like a fine guy. I’m just attacking some of his less well thought out ideas.

By: CS

CS — Sun, 30 Dec 2007 14:02:07 +0000

Mathematica attempts to simplify expressions in several functions. In this case, it simplifies (x^a)^b to x^ab before evaluating. It does this to shorten evaluation time for such functions as plotting and integrating. If you do not want expressions to be simplified, look into using Hold[] or the Compiled->False option. (Also, from the documentation for Sqrt and Power, it is stated that Power[Sqrt[x],2] is simplified to x.)Besides, the apparent "bug" here is that the given expression should be undefined if intermediate states are restricted to the real numbers. Mathematica handles complex numbers, in which case there is no problem with the expression. How is that a "bug"? What, is complex analysis not yet considered a valid branch of mathematics?The current behavior is explained in the documentation. Open source would be great, but if people can't be bothered to read the documentation, how can we expect that they will actually parse through the code whenever something unexpected happens?

By: The Real Anti-Ron

The Real Anti-Ron — Sun, 30 Dec 2007 09:44:45 +0000

Ron, smart of you to go deeper into the argument but if you go one level deeper you get this:

Most of the documentation provided for Mathematica is concerned with explaining what Mathematica does, not how it does it. But the purpose of this page is to say at least a little about how Mathematica does what it does. Some Notes on Internal Implementation gives some more details.You should realize at the outset that while knowing about the internals of Mathematica may be of intellectual interest, it is usually much less important in practice than one might at first suppose.Indeed, one of the main points of Mathematica is that it provides an environment where you can perform mathematical and other operations without having to think in detail about how these operations are actually carried out inside your computer.

By: Ron

Ron — Sun, 30 Dec 2007 08:48:00 +0000

If you go back and read Dr. Laden’s argument, it is partially based on the following problem cited in David Joyner and William Stein, the paper at the end of the links if you follow them. Maybe instead of attacking Dr. Lade you should read the article he is referring to:–begin quoteSuppose Jane is a well-known mathematician who announces she has proved a theorem. We probably will believe her, but she knows that she will be required to produce proof if requested. However, suppose now Jane says a theorem is true based partly on the results of software. The closest we can reasonably hope to get to a rigorous proof without new ideas) is the open inspection and ability to use all the computer code on which the result depends. If the program is proprietary, this is not possible. We have every right to be distrustful, not only due to a vague distrust of computers but because even the best programmers regularly make mistakes.–end quote —

By: Ron

Ron — Sun, 30 Dec 2007 08:42:43 +0000

A commentor named Anonymous is telling the writer of this post that he can’t “look under the hood.”