Attached below is a paper I wrote with Uli Doraszelski showing that continuous-time formulations of common IO dynamic games will be orders-of-magnitude faster than the popular discrete-time versions.
The paper was recently rejected by RAND Journal.
One major problem was that Haile and his referees claimed that the advantage of our continuous-time model was that it was a sequential move game, and wanted us to “admit” this. We refused because the game was, in fact, a simultaneous-move game, just like many other continuous-time games such as wars of attrition and innovation races.
After seeing that our revision did not buckle under to their demands that we make false statements about the game, the referees and Haile were unhappy. A referee claimed that our paper has a misleading “internal message .. the less sophisticated RAND reader will take away from this paper after reading it quickly (which I assume most readers do)” and Phil Haile agreed, saying that our paper was “fundamentally misleading” even though the same referee said that a careful reader would not be mislead.
To put it more clearly, Haile and his referee called me a liar not because of anything I wrote but because “the less sophisticated reader” would come away with a false impression “after reading it quickly.”
I share this experience with you so that you can see the hostile attitude towards computational work at the elite journals. It may surprise you that RAND would have this attitude given that it published Pakes-Maguire. This is easy to explain: I published Pakes-Maguire when I was a Co-Editor, and the current Editorial Board is, as revealed in this case, not supportive of research on computational methods.
I have had similar experiences with other major journals, such as Econometrica and the Review of Economic Studies.
I always tell my students that they need to be aware of this. My basic advice is to downplay contributions to computational methodology in their papers, particularly their job market paper. I think that computational research should be written up in a paper separate from the job market paper, and then have the job market paper cite the computational paper. If you try to introduce a computational method inside an application paper, editors and referees will tell you to cut the computational discussion down to the point where few will be able to see the computational idea.
I do not mean to discourage you. I want you to use the best numerical methods in your work and I believe that they will be more reliable and helpful than the standard methods used in economics. You just need to be aware of the hostility and ignorance that currently control the major journals and most of the top departments.
Below are the opening paragraphs of an e-mail I sent to all members of the RAND editorial board regarding Haile’s decision.
The Editorial Board took only 36 hours to respond by endorsing Haile’s handling of the paper.
Dear [Editor]:
Recently Uli Doraszelski and I received what was essentially a rejection letter from Co-Editor Haile regarding a revision of a submitted manuscript.
The manner in which this submission was handled was, in my opinion, the most outrageous treatment from an economics journal I have ever encountered. For example, the editor and referees demand that we describe our game as one with sequential moves whereas the game we study is, and always has clearly been, one with simultaneous moves. We are called stubborn for refusing to do this. If you find this acceptable editing, there is no need for you to read any more.
The short version of our experience is as follows. The first round of referee reports were critical of the paper for not being clear about the nature of the game. Uli wrote a very polite and deferential response pointing out how the referees’ negative comments were based on a misreading of the mathematical nature of our model. He also carefully noted how our objective was not to create an encyclopedia of ways to avoid the curse of dimensionality but rather how if one makes the usual game-theoretic assumptions, the continuous-time version avoids the curse of dimensionality that makes the discrete-time version computationally difficult. The response from Haile and his referees was negative. There was no acknowledgment of their errors, and a demand that we rewrite the paper along the lines they suggested. Haile refers to the most recent revision of the paper as “fundamentally misleading”, echoing the repeated, but confused and unsubstantiated, attacks on our character in the referee reports. I was not surprised that Haile and the referees misread the paper because continuous-time games are not common today. However, the fact that they continue to attack us after Uli’s careful and respectful response indicates that for some reason Haile and his referees do not or cannot understand what we are doing in this paper.
Below I give a detailed accounting of the false attacks on our paper, but I first will point out some comments that clearly display the manner in which Haile and his referees have treated our submission. The following paragraph appeared in a referee’s report on the revision:
“I continue to believe that there is an important idea and contribution in this paper, but it is still marred by a rather misleading title and a way of writing that will suggest to all but the most careful readers is that “the way to break the curse of dimensionality is to formulate continuous time models instead of discrete time ones”. Here I am not quoting anything the authors say in their paper, but rather the internal message I think the less sophisticated RAND reader will take away from this paper after reading it quickly (which I assume most readers do).”
Note how he puts an expression in quotes even though that clause never appears in our paper. He then admits that he is not quoting anything we say. If we never said that then why are we criticized for saying it? Why does the phrase appear in quotations? He explains this by saying that some “readers” will think we said that. On the other hand, the referee agrees that if a person has read the paper, the only kind of person properly called a “reader”, he will know we never said this. The referee appears unable to come up with a logically consistent criticism. The clear implication of this referee’s comments is that we are supposed to write a paper to make sure that “less sophisticated” readers who don’t read the paper but instead “quickly” skim the paper do not come away with a misperception about “the internal message”. Furthermore, Haile’s letter to us regarding our revision implies that he fully embraces this view.
The bottom line is clear: we are not judged by what we write but rather by the perceptions of people who don’t read the paper. This is an absurd standard and one that can be used to reject any paper ever written since there is always someone who would get a false impression after reading small bits of a paper. This is only one small example of the manner in which Haile and his referees treated this paper.
He offers us one last chance. He says that if we “now agree with the referees and want to revise the paper accordingly, I will be willing the consider the revision. This will require some significant changes to the exposition and, probably, even to the title of the paper”. The problem is that “agreeing” with the referees would have us make statements that we believe are misleading. He goes on to say that if we submit another revision, he will reject it “unless there is unanimous agreement that you have adequately responded to the referees’ comments (from both rounds) and that the revised paper accurately conveys the findings.”
Therefore, Haile continues to embrace all the criticisms made by the referees in both rounds, and demands that we rewrite the paper as instructed to do by the referees in rounds one and two of their reports. This is particularly important so I will repeat it: Haile demands that we agree with the referee reports from both rounds. In case there is any doubt about this, he says that he is giving veto power to the referees, effectively handing to them his responsibilities and authority as Editor. My responses to the referee reports makes it clear why we will not buckle under to the demands made by the referees. Editors often get crazy referee reports, but Haile makes it clear that he fully endorses all the demands and criticisms of the referees in both rounds. Therefore, as I describe the referee reports remember that any errors and expressions of hostility in the referee reports are fully embraced by Haile.
Haile does make some conciliatory comments in his letter. He even asserts that he has “a great deal of respect for” my work, a claim that has no credibility given the obvious contempt for this work. It is irrational for us to put any weight on friendly comments given the hostility and prejudice expressed elsewhere, particularly when he says that we must “agree with the referees”. Haile has clearly embraced all of the comments of the referees. If he has so little understanding of computational issues that he has to be a slave to the referees’ comments then he had no business handling this paper. At the time this paper was submitted, there were other Co-Editors who had worked on dynamic games. In particular, Jennifer Reinganum’s early work used continuous-time games similar to the one we present. Therefore, RAND Journal cannot claim that the problems in this case are due to a lack of expertise on continuous-time games.
The facts are clear: no one contests the originality of the work, no one disputes our claims that solving these games in continuous time increases solution speeds by orders of magnitude, no one disputes the importance of the stopping rule analysis, and no one disputes the mathematical correctness of any claim we make. Instead, the dispute is one of exposition and interpretation. They want us to make vague and imprecise statements instead of the mathematically precise ones we make.
My initial inclination was to just regard this as another example of the mindless hostility economists have towards computational methods, conclude that RAND Journal was no exception to this, and discourage my collaborators and students from considering RAND as an outlet for computationally intensive work. However, RAND was supportive of computational work in the past. Perhaps this is just a case of one bad apple.
Therefore, I decided to contact the Editorial Board to see what the general attitude was at RAND before attributing Haile’s attitude to RAND Journal. Uli and I want to move on, but I will delay submitting this paper to another journal for a short time so that you and the other Co-Editors of RAND Journal can consider the issue.
I also want to make clear that this is a letter from me. Uli had no hand in writing this letter nor was he consulted about it. I write this letter knowing that there is a strong likelihood of retaliation, but I strongly recommend that any retaliation be directed only at me.
After you read my response to the referees and editor, I ask you the following simple question: Does RAND Journal repudiate the attacks on our work and character made by Haile and his chosen referees and reverse his rejection decision, or does RAND Journal stand by Haile and endorse his treatment of this submission?
Sincerely,
Dr. Kenneth L. JuddPaul
H. Bauer Senior Fellow
Hoover Institution