Penn State 2021, March 18: MPEC, NFXP, Algorithms and Software

Return to Ken Judd’s 2021 Penn State Numerical Methods Course Home Page

March 18: MPEC, NFXP, Algorithms and Software

I will describe a variety of software and hardware issues regarding structural estimation.

(I use the European dating scheme — year-month-day — in my file names. This also means that they appear in proper time order below.)

The slides for my presentation are in “00 MPEC NFXP and future of structural estimation.pdf” which is the first pdf file attached below.

Here are some comments abut the evolution of Su-Judd and some criticisms.

Evolution of Su-Judd

The papers attached represent the evolution of the Su-Judd paper on MPEC. The first papers refers to the method as the Judd-Su method, but after the initial draft, Che-Lin Su did all the work so I made the change.

March 2, 2006: The very first draft. Presented at Northwestern University and University of Chicago in March, 2006

April 12, 2008: Original submission to Econometrica

November 14, 2008: R&R from Econometrica

Newey asks for an example where we apply MPEC to estimating a game. He points to referee 4’s report, which was by Rust who signed the report. So Che-in and Rust spent substantial time formulating and solving the game example that is in the revision. Newey also asked whether NFXP and MPEC produce the same estimates.

November 14, 2010: Revision and response to Editor’s comments

July 18, 2011: Decision letter

Newey tells us to drop the section which shows that NFXP and MPEC produces the same estimates. He also told us to delete the game example and forbad us from putting it in the online appendix. He says that some of his students told him that they found it difficult to apply the MPEC method.

Che-Lin received the decision letter during ICE11. I remember him coming down to the lecture room and telling me he had received the letter. He did not want to discuss it in the lecture room in front of the ICE11 students, so we went upstairs to the cafeteria where he could tell me in a more private area. He told me what Newey demanded, I thought about it for a few seconds, and I said — calmly — we should take the offer if that was what Che-Lin wanted to do. I think he was surprised by my reaction.

September 2012: Published version of Su-Judd

September 2012: Published version of Dube-Fox-Su

I include the Dube-Fox-Su paper because it was an immediate extension of Su-Judd. As part of our initial work, Che-Lin applied MPEC to Nevo’s code for BLP. I remember Che-Lin telling me that he found a different point estimate and it had a higher likelihood than Nevo’s result. As is often the case, he was happy to tell me this but then I asked another question: “Does the Nevo estimate lie inside the confidence interval defined by your estimate?” He went to work and told me “Yes.” We wanted to make sure we understood Nevo’s code, so, on that trip to Northwestern (he was a postdoc there at the time) we met with Nevo and asked him some questions about his code.

After that, he became acquainted with Dube and Fox. I taught my computation course in the Autumn quarter at Chicago at that time, so Che-Lin would visit me and then also met Dube and Fox (who were also at Chicago). They then worked on their paper applying MPEC to BLP, and submitted it to Econometrica. Steve Berry handled that paper.

At some time before we sent in our revision, Econometrica contacted Che-Lin and suggested that Su-Judd and Dube-Fox-Su be merged into one paper. He discussed this with Dube, who was against the merger. He did not tell me about this suggestion from Econometrica because he knew my opinion and was afraid of what it would do to my blood pressure.

At least three Econometrica Editors were involved in handling each of these papers.

Berry and Conlon Criticisms

In 2009, Steve Berry and his student, Chris Conlon, criticized MPEC.

Berry (who flew 3000 miles to give a >20-minute discussion of Dube-Fox-Su at a NBER meeting at Stanford, but also commented on Su-Judd) claimed that one “can think of MPEC as adding incidental parameters.” He also referred to the solvers we used as “magical black boxes” and asserted that there were “no proofs of the superiority of MPEC.” I also recall (but did not write in my notes) that Berry said that there is no convergence theorem for MPEC. After the conference, I asked Berry for his slides, but he did not send them to me. Therefore, I have to rely on my hand-written notes and memories.

Conlon said “One drawback of the MPEC approach is that the addition of nuisance parameters means obtaining standard errors is no longer straightforward, which leads [Su and Judd] (2008) to recommend bootstrap-type procedures.”

I asked both to defend their criticisms with more precise discussion of the issues they raise. They refused. I point to these criticisms because many people were aware of those criticisms but have never heard my response. I have also heard of comments by economists which indicate that they do not know the difference between NFXP and MPEC.

Rust et. al Criticisms

I also include the more recent critique of John Rust and collaborators. Their criticisms illustrate how important computational details are when discussing alternative methods.

Their June, 2016, version claims that MPEC fails dramatically at solving some examples that NFXP solves easily. Che-Lin was a referee and he found that the reason for the difference. The dynamic programming equations in their examples were very ill-conditioned because they took beta close to 1. In their NFXP examples, they first applied a preconditioning method that transformed the DP equations into a well-conditioned system that then worked well with NFXP. In contrast, their MPEC examples used the original ill-conditioned equations. Che-Lin showed that both methods blow up with the ill-conditioned equations and both did fine with the well-conditioned formulations.

At the same time, Lee and Seo submitted another criticism of Su-Judd. It also was flawed, as I describe in my October, 2011, message to them. So, Econometrica asked the Rust gang to merge with Lee and Seo to write a criticism of Su-Judd. It was accepted on July 30, 2015, went through a couple of revisions of the acknowledgment footnote, and appeared the following January. Rust et al. have more recently presented their criticism at an Econometric Society organized conference. I attach the slides.

Their current claims of NFXP being faster are very misleading.