Previous work has shown that in English ungrammatical center embeddings are more acceptable and easier to process than their grammatical counterparts (Frazier, 1985; Gibson & Thomas, 1999). A well-known explanation for this preference for ungrammatical structures is based on working-memory overload: the claim is that the prediction for an upcoming verb phrase is forgotten due to memory overload, leading to an illusion of grammaticality (Gibson & Thomas, 1999). However, this memory-overload account cannot explain the recent finding by Vasishth, Suckow, Lewis, and Kern (2008) that in German no illusion of ungrammaticality occurs. We present a simple recurrent network model that can explain both the presence of the grammaticality illusion in English and its absence in German. We argue that the grammaticality illusion emerges as a function of experience with language-specific structures, not working memory constraints as argued for in Gibson and Thomas (1999).