Introdução

Luís Miguel Carvalho

O objeto e os objetivos da compilação

Os textos reunidos neste e-book apresentam vários estudos sobre o Programme for International Student Assessment – comumente reconhecido pelo seu acrónimo – PISA, enquanto instrumento de regulação baseado no conhecimento, isto é, como um dispositivo técnico, de elevada sofisticação, que se baseia num tipo particular de conhecimento, com vista à orientação, coordenação e controlo da ação social, no setor da educação. Simultaneamente, o PISA é tomado, nestes estudos, como um analisador de continuidades e mudanças nos processos de regulação política e social da educação.

Os cinco textos que integram esta compilação testemunham uma parte do trabalho que sobre o tema desenvolvi, ao longo da última década, individualmente e em colaboração com outros colegas do Instituto de Educação da Universidade Lisboa (IE-ULisboa), e que foi revertido para publicação, em revistas internacionais da área da educação (Discourse: Studies in the Cultural Politics of Education, Educação & Sociedade, European Journal of Educational Research, European Journal of Education). Os textos foram selecionados de modo a oferecer uma visão compreensiva sobre o trabalho de pesquisa realizado, a sua problemática e seus diversos focos: a produção, a circulação e o uso ativo de racionais, de resultados e de análises gerados no contexto do PISA.

A publicação desta compilação pretende favorecer a divulgação da pesquisa a um público interessado no PISA - ou, mais genericamente, por temas de política educativa e pela análise social dos fenómenos educativos -, que não tem acesso habitual a publicações editadas no estrangeiro e destinadas a comunidades especializadas. A publicação serve, também, para apoiar a formação de alunos de pós-graduação em Educação/Ciências da Educação, especialmente no âmbito dos estudos sobre as políticas educativas e os processos de regulação da educação.

A perspetiva adotada nos estudos

Os estudos aqui apresentados não tratam, como porventura mais frequentemente se esperará, nesta matéria, de analisar e explicar ou de dar a perceber aos leitores os desempenhos dos alunos portugueses ou o ‘comportamento’ do sistema educativo nacional na prova coordenada pela OCDE. Tratam, antes, de observar o PISA sob uma perspetiva analítica que procura interpelar as relações entre conhecimento pericial e política, tendo em conta as seguintes tendências de mudança, percebidas nos modos de regulação da educação: o recurso mais intenso a ferramentas de governação baseadas no conhecimento; a complexidade decorrente da ampliação de escalas e de agências nos processos de regulação das políticas; e a preponderância das organizações internacionais nas dinâmicas cruzadas de regulação que daí resultam, uma preponderância que é frequentemente obtida exatamente por via do recurso ao conhecimento especializado. Como participam estes dispositivos técnico-políticos nos processos de ordenação, coordenação e controlo do setor da educação, eis a questão mais ampla que atravessa esta série de trabalhos.

Na resposta a esta questão, os estudos aqui reunidos consideram a interdependência de múltiplos fatores, mormente: a diversidade de atores implicados e os diversos contextos/escalas da regulação (transnacional, nacional, local) da sua produção, trânsito e receção; os principais quadros cognitivos e normativos em presença (em acordo e/ou em disputa) nesse fluxo de recursos informacionais e de atores humanos; os modos de regulação subjacentes e postos em marcha; o impacto (percebido) nas práticas de atores de educação aos quais o PISA é dirigido ou que o tomam como recurso para a ação.

O contexto da produção dos estudos

A unidade dos cinco textos firma-se, portanto, numa problemática comum. Mas deriva igualmente de um contexto específico de produção de pesquisa em educação. Com efeito, a série de estudos está vinculada a uma ‘linha de pesquisa’ que, desde há cerca de dez anos, vem sendo prosseguida no IE-ULisboa, designadamente no âmbito das atividades de investigação e formação em Administração e Política Educacional, e que vem equacionando, de diferentes modos e com diversos objetos empíricos, o papel do conhecimento na construção e regulação das políticas públicas de educação (Barroso, Carvalho, Fontoura & Afonso, 2007; Carvalho, 2015).

Esta agenda emergiu no quadro da participação da equipa de investigadores do IE-ULisboa, sob a coordenação de João Barroso, no projeto de investigação europeu KNOWandPOL (2006-2011).1 Nesse âmbito, foram estudadas modalidades de acesso e de uso do conhecimento especializado por parte dos decisores políticos (e.g., Barroso, 2010, Barroso, Carvalho & Figueiredo, 2012), foram traçadas genealogias da decisão nas políticas públicas de autonomia e gestão das escolas (e.g., Barroso, 2011; Lopes, 2011; Menitra, 2011) e de educação sexual (e.g. Carvalho e Figueiredo, 2011; Figueiredo, 2011), e foram inquiridos os processos de regulação desencadeados por programas de avaliação nacionais – a avaliação externa das escolas (e.g. Afonso & Costa, 2011, 2012) – e internacionais, concretamente o PISA (e.g. Carvalho, 2011; Costa, 2011).2

Assim, os textos escolhidos mostram a concretização e simultaneamente são depósitos de uma agenda de pesquisa esboçada, em 2008 (Carvalho, 2008), em função do interesse partilhado por um largo grupo de investigadores europeus pela compreensão das circunstâncias, significados e processos sociais implicados na construção do PISA, a nível supranacional, mas igualmente na circulação e na apropriação dos seus produtos informacionais, em contextos de ação situados à escala nacional. 3

Estudos em preparação

Não obstante a sua ligação a uma trajetória passada, a edição deste e-book tem uma outra ‘razão de ser’, a de apoiar – e de sinalizar – a emergência de uma nova vontade investigativa. Não se tendo esgotado a relevância ou a pertinência de ‘olhar’ as avaliações internacionais de larga escala, à luz do conceito de instrumento de regulação, muito menos desvanecido o nosso interesse investigativo sobre o PISA – e sua participação na regulação da educação –, pretendemos, nos próximos anos, inscrever os estudos sobre tais dispositivos num novo projeto coletivo de pesquisa (Carvalho, 2017): trata-se, agora, de tomar o PISA como analisador do fenómeno da mobilização de conhecimento especializado, suas modalidades e efeitos, no âmbito dos processos de coordenação e controle de sistemas educacionais.

Para o fazer, pretende-se investigar, em profundidade, ideias, processos e atores que interligam o conhecimento especializado e os contextos de elaboração de políticas, a administração da educação e o debate público, em Portugal, a partir de quatro eixos de inquirição: tradução – estudando a transformação dos resultados do PISA em conhecimento para as políticas públicas nacionais por atores estatais e não estatais; cobertura mediática – estudando a agenda e a divulgação do PISA pela imprensa escrita; aprendizagem – estudando o uso do conhecimento PISA pelos políticos, no contexto de sua ação situada; internalização – estudando a apropriação do conhecimento avaliativo associado ao PISA pela administração e por escolas envolvidas (nos ciclos PISA). Em conjunto, interessa-nos conhecer os modos como o conhecimento posto em circulação pelo PISA é transformado pelos contextos sociocognitivos que o recebem e eventualmente o redifundem, mas também os modos como tal conhecimento molda os próprios atores e contextos que o transformam.

Organização e conteúdo do livro

O e-book está organizado em três partes. Na primeira parte, apresentam-se os principais elementos do quadro concetual mobilizado na abordagem ao PISA, no âmbito do projeto KNOWandPOL, e prosseguido em estudos realizados já depois da sua conclusão. Recorre-se ao texto The fabrications and travels of a knowldege-policy instrument, publicado, pela primeira vez, em 2012, no qual sobressaem dois pilares teóricos fundamentais: a perspetivação do PISA como instrumento de uma regulação baseada no conhecimento; a observação dos processos de construção de conhecimento para a política como práticas cognitivas e sociais interdependentes e inscritas em dinâmicas de regulação transnacionais, nacionais e regionais/locais. Em complemento, o texto põe em destaque algumas das principais linhas interpretativas emergentes dos estudos conduzidos no projeto.

Na segunda parte da coletânea, inserem-se dois trabalhos publicados na revista Educação & Sociedades, nos quais se aborda o tema da regulação transnacional da educação tendo por base a análise da ação da OCDE no âmbito do PISA. O texto Governando a educação pelo espelho do perito (publicado em 2009) equaciona o PISA, em suas atividades de inquirição, organização e publicação, como um complexo processo de construção de um espaço de regulação política transnacional, dirigindo a atenção do leitor para os elementos cognitivos e normativos do PISA relacionados com a definição da ‘realidade educacional’, com a determinação das formas ‘apropriadas’ ao seu governo e com a produção de conhecimento para a política. Já o segundo artigo – Intensificação e sofisticação dos processos da regulação transnacional em educação: o caso do PISA –, de publicação mais recente (2016), apresenta uma síntese sobre os processos e os efeitos das dinâmicas de regulação transnacional baseada no conhecimento, no contexto do PISA, mas como o título procura sinalizar, incide na observação de novas dinâmicas de alargamento da influência do PISA, evidentes, em anos mais recentes, por via de novos modos de produção, de troca e de difusão de conhecimento pericial.

Finalmente, a terceira parte inclui dois trabalhos que versam as apropriações do PISA, nos contextos nacionais, e que resultam da colaboração com outros autores. O artigo Seeing education with one’s own’ eyes and through PISA lenses, submetido em 2014 e publicado em 2015, elaborado em parceria com Estela Costa, dá conta de processos de reinterpretação e de recontextualização que marcam a circulação dos resultados do PISA, ao longo de seus três primeiros ciclos (200,2003,2006), em seis espaços europeus (Bélgica francófona, França, Hungria, Portugal, Roménia e Escócia). O texto dá especial atenção a fenómenos de ‘atualização das sociedades de referência’ (i.e. as sociedades tomadas como referências para a reforma dos sistemas educacionais) no contexto das receções ativas do PISA, analisando-as como parte de um processo compósito que envolve razões domésticas (relacionadas com as agendas para a educação ou com fatores históricos profundos) e injunções cognitivas e normativas introduzidas pelo PISA. Já o artigo Fifteen years looking at the mirror, elaborado em colaboração com Estela Costa e Catarina Gonçalves, e publicado no início do corrente ano (2017), centra-se apenas no contexto português e desenvolve-se em torno de uma questão principal: o que fazem os atores das políticas públicas com os dados e as análises PISA, em momentos nos quais expressam e/ou justificam as suas escolhas? O estudo faz sobressair dois fenómenos principais relacionados para a trajetória da receção do PISA, em Portugal, entre 2000 e 2015: a consolidação da sua credibilidade como fonte para processos e textos políticos; o surgimento de novos atores que pretendem traduzir os resultados do PISA em conhecimento para a política nacional, após um longo período de inexistência dessa intenção. Em conjunto, estas tendências permitem fazer notar o papel regulador do PISA, nos processos de política portugueses, mas igualmente mostram a contribuição relevante das políticas de receção na legitimação desse papel.

Referências

Afonso, N., & Costa, E. (2011). A avaliação externa das escolas: Um instrumento de regulação baseado no conhecimento. In J. Barroso, & N. Afonso (Org.), Políticas educativas (pp. 155-189). V. N. Gaia: Fundação Manuel Leão.

Afonso, N., & Costa, E. (2012). Les politiques d’évaluation des écoles. Le cas du Portugal. Recherches sociologiques et anthropologiques, 43(2), 53-75.

Almeida, A. P. (2015). Atores, regulação e conhecimento nas políticas públicas de construções escolares em Portugal: As Escolas de Área Aberta. Tese de Doutoramento. Lisboa: ULisboa - Instituto de Educação.

Barroso, J. (2010). Conhecimento, actores e política. Sísifo: Revista de ciências da educação, 12, 37-50.

Barroso, J. (2011). Conhecimento e acção pública: As políticas sobre gestão e autonomia das escolas em Portugal (1986-2008). In J. Barroso, & N. Afonso (Orgs.), Políticas educativas (pp. 27-58). V. N. Gaia: Fundação Manuel Leão.

Barroso, J., Carvalho, L. M., & Figueiredo, C. C. (2012). Knowledge and policy configurations: An interpretative analysis drawing on sex education policies (Portugal, 1984-2009). In J. Barroso, & L. M. Carvalho (Eds.), Knowledge and regulatory processes in health and education policies (pp. 51-88). Lisboa: Educa.

Barroso, J., Carvalho, L. M., Fontoura, M., & Afonso, N. (2007). As políticas educativas como objecto de estudo e de formação em administração educacional. Sísifo: Revista de ciências da educação, 4, 5-20.

Carvalho, L. M. (2008, março). Guidelines for field research [KNOWandPOL, Education Sector] - Production of OECD’s “Programme for International Student Assessment”. Documento não publicado.

Carvalho, L. M. (Coord.). (2011). O espelho do perito: Inquéritos internacionais, conhecimento e política em educação. V. N. Gaia: Fundação Manuel Leão.

Carvalho, L. M. (2015). As políticas públicas de educação sob o prisma da ação pública: Esboço de uma perspetiva de análise e inventário de estudos. Currículo sem fronteiras, 15(2), 314-333.

Carvalho, L. M. (Coord.). (2017, maio). Aprendendo com o PISA: Mobilização de conhecimento especializado e regulação da educação. Projeto apresentado ao concurso Projetos de IC&DT em todos os Domínios Científicos 2017 (Fundação para a Ciência e a Tecnologia). Documento não publicado.

Carvalho, L. M., & Figueiredo, C. C. (2011). Configurações de conhecimento e política na regulação da educação sexual em meio escolar. Educação, sociedade & culturas, 34, 67-87.

Carvalho, L. M., Viseu, S., & Gonçalves, C. (no prelo). Bridging worlds and spreading light? Intermediary actors and the translation(s) of knowledge for policy. In C. E. Mølstad, & D. Pettersson (Eds.), Fear of the dark: The quest for educational knowledge appearance (Routledge)

Costa, E. (2011). O Programme for International Student Assessment (PISA) como instrumento de regulação das políticas educativas. Tese de Doutoramento. Lisboa: ULisboa - Instituto de Educação.

Cruz, C., & Carvalho, L. M. (2017). A relação entre conhecimento e políticas locais de educação. In C. A. Gomes et al., (Org.), Atas do XIII Congresso SPCE: Fronteiras, diálogos e transições na educação (pp. 1347-56). Viseu: Escola Superior de Educação de Viseu/Sociedade Portuguesa de Ciências da Educação.

Figueiredo, C. C. (2011). Redes sociais e políticas: Genealogia das políticas públicas de educação sexual. Tese de Doutoramento. Lisboa: ULisboa - Instituto de Educação.

Gama, A. (2017). Peritos, conhecimento e regulação da educação: Estudo da ação dos peritos externos no Programa Territórios Educativos de Intervenção Prioritária, entre 2006 e 2012. Tese de Doutoramento. Lisboa: ULisboa - Instituto de Educação.

Lopes, J. H. (2011). A contratualização como instrumentação da acção pública. In J. Barroso, & N. Afonso, (Orgs.), Políticas educativas (pp. 87-121). V. N. Gaia: Fundação Manuel Leão.

Menitra, C. (2010). Autonomia e gestão escolar: Os debates parlamentares em Portugal (1986-2008). In J. Barroso, & N. Afonso (Orgs.), Políticas educativas (pp. 59-85). V. N. Gaia: Fundação Manuel Leão.

Menitra, C. (2013, September). Knowledge and public action: Mapping the ‘worlds’ of new political actors in education policies. Paper apresentado na European Conference on Educational Research 2013, Istambul.

Viseu, S., & Carvalho, L. M. (2017, August). Think tanks, policy networks and educational policy: An exploratory study about EDULOG. Paper apresentado na European Conference on Educational Research 2017, Copenhagen.

1 Ver www.knowandpol.eu

2 Trabalhos mais recentes deram – e ainda dão - continuidade a esta agenda, nomeadamente equacionando o papel do conhecimento pericial nas políticas de políticas de construções escolares (Almeida, 2015) e de intervenção prioritária (Gama, 2017), o papel do conhecimento na construção de políticas locais de educação (Cruz & Carvalho, 2017), o papel de comissões e de grupos de trabalho na regulação da educação (Menitra, 2013), a emergência de atores intermediários (Carvalho et al., no prelo) e de think-tanks (Viseu e Carvalho, 2017) na intensificação da presença de conhecimento especializado na elaboração e/ou legitimação das políticas e da ação pública.

3 Trata-se de uma agenda de pesquisa que partilhei com outros colegas do IE-ULisboa com quem elaborei variados trabalhos (relatórios, artigos), sobretudo com Estela Costa, cuja tese versa exatamente o estudo do PISA como instrumento de regulação (Costa, 2011), mas igualmente com Natércio Afonso. Os textos que aqui reúno incorporam, direta ou indiretamente, suas relevantes contribuições ao longo desse percurso. Para aceder a alguns trabalhos realizados sobre o PISA no âmbito do projeto KNOWandPOL por investigadores de outros países, consulte-se o dossiê temático ‘O PISA e as políticas públicas de educação’, que coordenei para a Sísifo - Revista de Ciências da Educação (10, 2009).

The fabrications and travels of a knowledge-policy instrument 1

Luís Miguel Carvalho

Introduction

On the morning of 8 December 2010, somewhere in the European skies, a passenger looks at the front pages of two newspapers. There it is: ‘OECD [Organisation for Economic Cooperation and Development] report: the Portuguese are the students who have made most progress’ (O Público) and ‘France, the country of large school disparities’ (Le Monde). Interested, the traveller begins a quick online search on his laptop – and there it is again, and again: ‘PISA [Programme for International Student Assessment] literacy survey: Finland loses top spot to Asians’ (BBC News Online); ‘Our school is progressing, but it is still very average ... We stay in the pack’ (Le Soir – a newspaper of the Belgian francophone community); ‘The Romanian PISA results are as bad as they were in 2006’ (Edumanager.ro – a Romanian newscast site); ‘PISA test: why did the literacy skills of Hungarian pupils increase?’ (HGV – a Hungarian newspaper); and ‘Commission welcomes findings of international student survey (PISA) – European results improve’ (European Union website).

‘Hmm ... what a ubiquitous thing’, he ponders. During the last decade, the Programme for International Student Assessment, carried out under the auspices of the OECD, has been present in newspapers and other media formats, at least triennially, in accordance with the periodicity of the survey. However, the media is not the exclusive carrier of the PISA results or the only way through which it comes into play in the educational sector, and the ‘league table’ rationale is not the only form that supports its travel to – and through – diverse educational contexts. Actually, several products generated with the PISA label (reports, methods and tables) have been, and are being, (differently) used in diverse social spaces (academic, bureaucratic and political) and scales (local, national and regional). From the making of secondary research to the reflection on education systems, and from the making of assessment tools based on the PISA rationale and method to the mobilisation of PISA to galvanise disputes or to legitimise policies, the acronym seems to pervade educational knowledge and policy contexts.

But PISA is more than an unavoidable acronym of the current educational lexis. It is part of a universe of knowledge which, paraphrasing Lindblad and Popkewitz (2004, pp. xx-xxi), ‘ensures’ that expert-based education policies can lead each nation into the so-called knowledge society. The articles included in this special issue discuss the OECD’s self-portrayed ‘quality monitoring tool’ as a device that embraces and conveys different ways of imagining (and doing) education, schooling and social research, and that – simultaneously – plays a part in the coordination of education policies and public action. PISA is here perceived as a knowledge-based (and knowledge-generating) regulatory tool (Pons & van Zanten, 2006; Freeman et al, 2007). Having in mind the concept of a public policy instrument (Lascoumes & Le Galès, 2004; 2007, p. 4), one might say that PISA is driven by a specific ‘problematisation’ of the role of education in contemporary times and by a specific model for the regulation of the educational sector. From this perspective, PISA displays a particular way of challenging education systems and national public policies, and, concomitantly, bears principles and practices for the making of (transnational) policy regulation based on expert-based mutual surveillance practices supplied by international organisations. Thus, this special issue also takes on the idea that changes in the making of knowledge (for example, research questions or aims partially defined by others rather than the researcher) and changes in the ways of governing (for example, the sophistication of policy instruments supported by the ‘generation of governing knowledge’) are creating a ‘closer interdependency’ between policy and knowledge (Ozga, 2008).

In order to clarify the approach followed in this collection of articles on the fabrication and circulation of PISA, I need to emphasise, from the very beginning, that the programme is not merely a triennial survey or report. Together with the inquiry, relevant face-to-face activities take place (in meetings, workshops, seminars, etc.). Likewise, multiple publications – apart from the survey’s main reports – are generated and have a worldwide flow. All these activities involve a great variety of social worlds – public and private research centres, experts, OECD professionals, policy makers, bureaucrats and technicians from many countries – and multiple kinds of knowledge, interests and perspectives. The programme is, from this point of view, a complex of activities, objects and actors that generates diverse resources for social action in various social spaces. Thus, our purpose is to analyse the ‘n-moves’ of PISA. Generated and driven by the OECD, and transnationally fabricated by individuals and organisations from different geopolitical and knowledge spaces, PISA’s objects circulate through national, regional and local public action settings, where different social groups have interests in them and, importantly, are using them differently.

This article sets out the main elements of the conceptual framework for the overall approach to PISA taken in this special issue. The first section develops three pillars of our schema (introduced in summary form in the previous paragraphs): the representation of PISA as a knowledge-policy instrument; the understanding of the making of knowledge for policy as a process that entwines cognitive and social practices; and the idea of a multidirectional construction of knowledge-policy instruments. The second section – mobilising analytical ingredients from a study on the production of PISA at the supranational level (Carvalho with Costa, 2009) – provides an extended look at the organisational and cognitive dimensions of the transnational fabrication and circulation of PISA. It pays closer attention to the process of gathering and coordinating social worlds around the manufacturing of the instrument, to the plasticity of knowledge for policy, and to the multidirectional and multivocal making of the tool and the fictions added (and/or remade) throughout PISA’s circulation through different discursive spaces. In conclusion, the article addresses an (apparent) paradox: the convergence (of multiple social worlds) towards PISA (object and processes) goes hand in hand with a conspicuous divergence in the use of the knowledge-policy instrument.

Studying the Fabrication and Circulation of PISA

A Knowledge-Policy Instrument

Officially launched as an OECD ‘response’ to member countries’ ‘need’ for ‘reliable’ data on their pupils’ skills and on the performance of their education systems, PISA’s ‘chief aim’ is, according to the organisers, to provide ‘a stable point of reference against which to monitor the evolution of education systems’ (OECD, n.d., p. 17). With this overt policy orientation, an explicit vision about the ‘policy–knowledge’ relationship and knowledge-generation practices is put forward: it is ‘a collaborative effort, bringing together scientific expertise from the participating countries, steered jointly by their governments on the basis of shared, policy-driven interests’ (OECD, 2001, p. 3).

However, PISA is more than a new (or a developed) type of international comparative assessment created in order to assist policy-making with ‘robust’ data and/or information. For us, it is a knowledge-policy instrument. To repeat, this means that PISA is a knowledge-based and knowledge-generating device which, combining technical and social components, takes part in the setting up of rules for the coordination and the control of public action in education. This is the first meaning I ascribe to the word ‘fabrication’ – PISA is both ‘fiction and making’ (Popkewitz, 2000), and operates over several core categories of schooling (to a certain extent rewriting the educational model of contemporary societies), redefining students as lifelong learners, redefining teaching–learning relationships and settings, and redefining school knowledge. And it also sustains particular cultural dicta about policy makers and policy-making in contemporary times. I outline just two examples in the following paragraphs (for an extended description and discussion, see Carvalho, 2011).

The innovative focus that PISA claims to bring to assessments – ‘rather than examine mastery of specific school curricula, PISA looks at students’ ability to apply knowledge and skills in key subject areas and to analyse, reason and communicate effectively as they examine, interpret and solve problems’2 [1] – constitutes a double shift. It is a move away from the self-reflection of conventional school systems based on their own categories, and from the rationale of previous comparative studies of students’ performance and their dependency on national curricula. But parallel with this differentiation runs a redefinition of the appropriate school knowledge for the ‘knowledge society’. Following Mangez (2008, pp. 102-104), the notion of competence enacts a utilitarian perspective on knowledge as it takes practical usefulness in solving everyday problems as the main criterion for the assessment of school knowledge. Therefore it promotes a restructuring of curricular composition not in conventional disciplinary terms, but in trans- or cross-disciplinary ones. These shifts go hand in hand with calls for change in teaching and learning structures (for example, from ‘hierarchical’ to ‘organic’ models or from the sequentially transmitted ‘bodies’ of knowledge to the construction of learning by means of students’ cognitive connection with what they already know).3

School systems have to adapt to changes in the economy and society from industrialisation to post-industrialisation knowledge economies and societies – this is a recurrent topic in PISA documents. The programme incorporates and disseminates ways of thinking and doing that are supposed to enable policy makers to ‘naturally’ take part in the exercises of monitoring and mutual surveillance (practices that are seen as appropriate for – and thus defining – their identities and roles). The arguments developed in the OECD texts or by the OECD’s executives and/or collaborators (see, for example, OECD, n.d.; Adams, 2003; Schleicher, 2007; Turner, 2007; Hugonnier, 2008; Weinberg, 2008) assert that PISA supplies policy makers with rigorous comparative data, which thus allows them to conduct themselves rationally; learn about the place of each educational system in the worldwide competitive space; identify their country’s relative positioning vis-à-vis the ‘knowledge society’ and/or ‘knowledge economy’; and, consequently, ‘move’ their education systems to the (physical and symbolic) time of ‘tomorrow’s world’.

The Making of Knowledge for Policy Entwines Cognitive and Social Practices

As a consequence of this, PISA is explicitly oriented towards the production of data, information and knowledge for the steering of educational systems. Knowledge for policy’ is a specific type of knowledge, created by specific practices and referencing specific criteria (see Nassehi, 2008). In the making and flow of this type of knowledge, it is not only the quality of scientific or technical ‘credibility’ that counts; the criterion of ‘contextuality’ also counts – that is, following Lindquist (1990, pp. 31-35), the quality or condition of being relevant and able to be handled by an audience.

This means that the creation and the storing of data for others or the transformation of data into arguments is only relevant or has value if it is ‘usable’ by the desired or expected readers. If the success of knowledge for policy rests on the condition of being consumed, shared or learned by its audiences, then it cannot be separated from processes of ‘knowledge mediation’ (see, for example, Henry et al, 2001; Rutkowski, 2007). Such processes include the building of common understandings about what counts as knowledge for policy and the modes and content entailed, and about education and the ways it can or should be governed. Such ‘mediation’ includes processes of communication and organisation (i.e. the stabilisation of relations of cooperation) between different social universes.

As I noted earlier, PISA is not merely a triennial survey with a subsequent triennial report, but a complex of activities – ‘information gathering, publication and convocation activities’ (Lindquist, 1990, p. 34) – where communication and organisation happen. Figures 1 and 2 help to illustrate the point. The formal portrait of PISA participants (Figure 1) reveals researchers, public and private research centres, experts in the material assessed and in the components of scientific surveys, OECD professionals (either managers or analysts), policy makers, bureaucrats and technicians from multiple countries.4 These actors get together through multiple activities. Figure 2 (regarding the making of the ‘framework’ for PISA 2003) gives a representative example of the flow of exchange activities (for example, meetings and workshops) and the connected activities (for example, the preparation of frameworks and other documents or inquiry-related issues).

Figure 1: Key formal participants in PISA (Turner, 2007).

In considering these elements, we suggest that the making and circulation of PISA intertwines (1) a cognitive dimension – regarding the definition of what counts as knowledge for policy (and the modes and content entailed), as well as a vision of education and the ways it can be governed – and (2) a social dimension – concerning the creation and stabilisation of the process of organisation. In other words, the definitions of educational reality and appropriate conduct of government (and even of appropriate ways for the making of knowledge for policy) go hand in hand with the process of achieving coordination and cooperation between the different actors involved. Thus, the accomplishment of PISA depends on bringing together – and ensuring the cooperation of – various actors around a flow of activities, and on having them share the vision of PISA as a proper instrument for the governing of education.

Figure 2: The sequence of meetings related to the making of the PISA 2003 framework (Costa, 2010).

When we try to analyse the meanings and the scripts displayed within and around the knowledgepolicy tool, we need to keep in mind that – following Bruno Latour (1996) – these are the result of ‘several interdependent practices’: from the construction of research subjects to the institution of a ‘monopoly of competence’; from the convincing of others (politicians, bureaucrats, experts and researchers) in order to guarantee resources (informational, financial and human), to the building of ‘public confidence’. Thus, we follow closely some central elements of the ‘sociology of translation’ as they have been mobilised in the analysis of public policies – namely, the overlapping of cognitive activities and interactions between actors, as well as the importance given to physical elements (objects such as documents and databases) as participants in the making and sustaining of ordered practices (see Lascoumes, 1996; 2004, p. 437). This is my second way of invoking the notion of fabrication – that is, as the cognitive and social making of a complex knowledge-policy instrument.

Being Attentive to the Multidirectional Flows of Knowledge and Policy

Such creation and exchange of knowledge for policy may well be related to the ‘construction of multilateral spaces’ by international organisations (see Rutkowski, 2007, p. 237). The influence of intergovernmental organisations in education policies has been growing since the end of the Second World War, keeping pace with the intensification of interdependence and coordination of human activities on a worldwide scale (see Henry et al, 2001; Rizvi & Lingard, 2009). While these organisations have broadened their ‘scope of action’ (influencing national debates and the aims, goals and structures for/of education systems), they have also spawned new arenas and new forms of educational governance (see Leuze et al, 2007, pp. 9-10). These processes have not necessarily eliminated the role of national states in the governing of education systems, but they have certainly forced them to make new decisions, such as those concerning the loci and agencies responsible for the coordination of action in the educational field (Dale, 1999; Lindblad et al, 2002).

PISA is here analysed as an exemplar of the agenda and of the specific regulation practices that characterise the OECD’s involvement in transnational governance (see Mahon & McBride, 2008, 2009).5 The work generated and coordinated by a ‘policy lender’ (Steiner-Khamsi, 2004) does not, however, obliterate the idea that public policies are processes made up of a complex set of actions and interactions carried out by multiple actors in multiple spaces. Moreover, the ‘international argument’ (see Schriewer, 2000) is often cited when drawing up national policies as a strategy for the certification (or decertification) of controversial policies, operating as an act of validation or dismissal of actors and opinions by using a source of external authority (Steiner-Khamsi, 2004, pp. 203-204).

Actors do not ‘produce instruments out of nothing’, rather they do it through work that involves the ‘translation, interpretation and appropriation of ideas, knowledge and devices already in circulation’ (Freeman et al, 2007, p. 5). We should bear in mind that many of the categories that circulate on a worldwide scale – and which carry the weight of authority of universal categories – have a local or regional origin (Popkewitz, 2000, 2003). One of the main ‘technical-scientific’ ingredients of PISA – the item response theory, or item response model, as its critics prefer to call it – is a good example. The construction of test items for PISA surveys follows a version that was developed by the Danish statistician Georg Rasch in the 1950s and which has been disseminated in the US academic world since the 1960s by Benjamin Wright (director of the Measurement,

Evaluation, Statistics, and Assessment Laboratory at the University of Chicago). The work of Wright was continued by his student Geoffrey Masters, who was to become the chief executive officer of the Australian Council for Educational Research, and subsequently carried out by a student of the latter, Raymond Adams, an academic expert in psychometrics and International Project Director for PISA (Morgan, 2007; see also Vrignaud, 2006).

Thus, I analyse the making of knowledge-policy instruments as being a part of multidirectional processes that involve reinterpretation, decontextualisation and recontextualisation, and where national, local, regional and international agencies intertwine (see, for example, Popkewitz, 2000; van Zanten & Ball, 2000; Steiner-Khamsi, 2004; Barroso, 2006; Maroy, 2006). Thus, I adhere to the advice of Jürgen Schriewer (2000, p. 327) regarding the need to analytically address a ‘weaving of opposites’: ‘internationalization and indigenization’, ‘supranational integration and intra-national diversification’, and the ‘global spread of standardized educational models ... and the surprising diversity of socio-cultural interrelationship networks’.

Therefore, we understand PISA as a complex of multiple activities in which multiple social actors are engaged – either in convergent or divergent ways, actively or passively – in the production, dissemination and use of knowledge for policy. The actors involved are simultaneously ‘free and networked’, ‘constructing and being constructed by their engagement’ (Lawn, 2006, p. 285). And, I might add, the enrolment and the engagement of such heterogeneous actors, ideas and interests is a central condition for the ‘success’ of the programme. In the following section, I try to go deeper into the outlined perspective by means of a selective mobilisation of (mainly) analytical elements from our empirical study of the supranational fabrication of PISA (Carvalho with Costa, 2009).

The Fabrication of a (Travelling) Object and the Fabrications on the Move

Scenario: pragmatic comparison – mutual surveillance – soft regulation

PISA is part of a long-term movement towards the institutionalisation of a practice of comparative knowledge and, concomitantly, of a particular script for knowledge-policy relationships. In brief, this movement takes as its key principles that: (a) regular and systematic assessments are truthful practices for the improvement of national education systems; (b) such improvement has to be analysed hand in hand with the pace of change of other countries; and (c) international comparisons of students’ performance are responsible for changing the concept of quality applied to education systems, getting to the core activity (students’ achievements), and, in addition, are also able to respect educational complexity, collecting relevant data on learning structures, socioeconomic backgrounds or policy choices.

PISA is part of a comparative knowledge project steered by pragmatic reasoning, committed to the building of indicators for decision-making and conceiving comparison as intrinsic to decisionmaking (see Nóvoa, 1998). Nonetheless, PISA reshapes the earlier knowledge tradition of comparative study. It has a set of characteristics that makes it different from other similar instruments and relatively more successful: the ‘frequency of tests’ (triennial) and ‘diachronic coherence’; the ‘focus’ on a narrow set of expertise; the ‘flexibility’ of the programme owing to its modular structure; the ‘stability of the population’ (15-year-old pupils); the use of ‘tests focusing on competencies and not on the school curriculum’; and the ‘politicisation of the programme’, ensuring that the funding governments have a say in controlling the priorities of the programme and its application (Bottani, 2006).

In order to understand PISA, we need to take the OECD – its resources, ideas, ecology and choices – as a significant (though not the only significant) element of the equation. First, PISA has been developed in the context of an OECD educational policy agenda marked – since the 1990s – by the idea of ‘monitoring quality’, and as part of an agenda where educational problems and solutions are engendered to face the so-called needs of the knowledge economy (Rinne et al, 2004, pp. 459-462). As the joint head of the OECD’s Education Department recently reminded us, from the agency’s point of view, ‘the statistics, international comparisons and identification of practices and policies that lead to good results’ are some of the ‘tools’ that the organisation develops to ‘help the political decision’ – developing these tools is, indeed, the OECD’s ‘role’ and PISA is one of the ‘means’ to carry this out (Hugonnier, 2008, p. 48).

Second, OECD intervention in transnational governance has certain particularities. It is an intervention through ‘soft modes of governance’ (with a focus on ‘surveillance of performances’ and ‘assessment of policies’) and has important impacts in national policies as a ‘creator, purveyor and legitimator of ideas’ (see Mahon & McBride, 2008, pp. 7-15). These social and cognitive features of the OECD’s agenda and modes of governance are performed in PISA. It is, in Bengt Jacobsson’s (2006) terms, a ‘norm and standard tool’ made through a set of soft regulation practices combining ‘meditative’ and ‘inquisitive’ activities. These activities are sets of practices in which experts are recruited (1) to examine modern state structures governing procedures and choices, and (2) to elaborate standards in order to constitute governing practices. What is more, in these activities, the national states voluntarily hold on to mutual surveillance practices in the form of regular surveys.

The Gathering and Coordination of Social Worlds

During the last 10 years, there have been multiple increases in PISA: the expansion of the geographical and political scope with the growing number of countries involved, from 32 in the first cycle to 75 in 2009 (and especially the number of non-OECD countries involved, from 4 to 41), and the expansion of the object – i.e. an increase in the number and focus of the surveys, and target populations. There is a new literacy domain (information and communications technology) and the expectation of assessing a new one (‘interpersonal skills’). There is the inclusion of younger age groups and samples by grade or class. By expansion of the object, I also mean the linkage made with other OECD programmes such as the Teaching and Learning International Survey, which focuses on teachers’ work environments, or the Programme for the International Assessment of Adult Competencies. A third form of expansion is the building of new partnerships (for instance, with new researchers or with teachers’ unions) and the broadening of the number of research conglomerates – including more state institutes, public and private universities, or research and development units – which are contracted to carry out specific parts of the survey, or to build specific tools or make specific thematic analyses. In sum, PISA expands by means of these (exchangeable) packages of knowledge – that is to say, it assembles researchers through the commodification of knowledge practices.

These developments are related to PISA’s ecology.6 On the one hand, the expansion cannot be separated from a cultural and political environment that disseminates the conceptions of ‘global economic competitiveness’ and the ‘knowledge economy’ (see Broadfoot, 2000). On the other hand, it cannot be understood independently from ‘the emerging age of accountability’ (Hopmann, 2007, p. 400). But while we recognise the OECD as capitalising on this environment, we also have to recognise the collection of practices that keeps PISA alive and expanding in a field populated by other agencies which ‘export’ educational monitoring devices and are involved in the making of usable knowledge-policy instruments for national or regional territories. Thus, PISA is a complex tool that is made to travel, be mobile and be portable by different actors – it is made to attract and attach people, to generate public debates and public policies. Therefore, we must be aware that the making of the tool is also the making of an organised space – the existence of PISA depends on the effective connection of interested (individual and collective) actors.

When we investigate who’s who in the making of PISA frameworks and the main technical scientific tools, we might say that the ‘monopoly of expertise’ created around PISA rests on a somewhat narrow social basis, which includes a nucleus of between 10 and 20 actors from private and public organisations, experts from different areas and OECD members.7 They participate in diverse ways in the coordination of PISA’s multiple activities – inquiry, exchange and publications – and through these activities they bring together many other different actors and agencies from within and outside the formal structures (other experts, bureaucrats, policy makers, journalists and researchers).

The OECD’s PISA Secretariat is just one of those net-makers, but a central one. And it takes seriously the OECD libretto, presenting itself as playing the ‘catalyst’ role: The image I had [from the ‘very beginning of PISA’] was the image of a catalyst ... the substance doesn’t matter, but the presence matters. Basically being able to get the right people to the room at a political level, at a scientific level, a technical level, let them negotiate, let them discuss. There have been times when it was difficult to find an agreement and then we play a part. (Carvalho with Costa, 2009, pp. 74-75; interview with a member of the OECD Secretariat)

When we combine interviews, records of board meetings and PISA reports, it is apparent that the role of the Secretariat goes well beyond being a facilitator or an accelerator of interactions between research and policy.

Performing as an independent provider of ideas (Marcusson, 2004; Mahon & McBride, 2008), the OECD is the contractor of a specific framework (the competences of literacy), generated on its own initiative and attributable to its own agency.8 It was with such a framework that it became possible to interest others in new ways of thinking and doing knowledge and policy. Moreover, the problem the catalyst has to solve is not only that of ‘bringing actors and ideas into relation’, but also that of ‘keeping them in’ and acting under specific scripts, and, simultaneously, fulfilling their informational needs. The issue is also how to coordinate those social worlds or which ‘coordination forms’ – that is, ‘formal and informal mechanisms and rules defining the parameters of social interaction’ (Delvaux, 2008, p. 26) – have to be performed. Having the formal role of ‘manager of everyday activities of PISA’, the OECD Secretariat is able to regulate the timings, resources and flows of the activities and relationships between the actors. Being a ‘manager’ also means ensuring that certain procedural rules are followed – for instance, the definition of positions and relations between formal PISA structures and actors along the several sequences of tasks that make PISA (such as preparing frameworks, testing instruments, implementing the survey, treating and discussing data, preparing publications – see Figure 2).

The key participation of the OECD Secretariat in the orientation and control of the major PISA meetings is a common element in the narratives of national representatives about the dynamics of the PGB meetings (see Carvalho with Costa, 2009, pp. 89-91). However, three other – related – ideas emerge from the stories. First, meetings are political spaces where conflicts between visions of education take place, and where compromises are established between participants that have unequal resources (for example, their nearness to or distance from the specific technical knowledge of PISA, their mastery of the English language or the unequal status of each country in the ‘political-economic ladder’ of the OECD) and unequal capacities to exercise those resources.

Second, the technical argument has a central role for the achievement of political consensus, as, for many, the (political) building of consensus seems to be carried out in the background of technical expertise. Third, the use of (written) consultation practices emerges as an effective strategy for reaching consensual decisions. The ways of producing an enormous amount of documents – mobilising people to work on successive draft versions – along the PISA flow of activities (see Costa, 2010) constitute a complex system of normative and functional coordination.9 These ‘open processes of writing’ generate perceptions of membership and belonging, and (at least) of partial authorship; and they effectively conduce to the production of consensual decisions (for example, regarding the framework or regarding the structure and writing style of the reports) and to the adoption of standardised procedures that steer the participants (for example, the technical norms that guide the implementation of PISA inquiries).

The ‘manager of everyday activities’ operates over functional and symbolic issues. In other words, the interactions between participants (experts, bureaucrats, policy makers and their representatives, researchers and OECD members) are regulated by functioning principles and methods that are expected to take place in a social space created and managed by an ‘expert organisation’ which must keep its credibility as an independent ‘truth-teller’ (Noaksson & Jacobsson, 2003, p. 42). Consensus-building, a high level of trust in expertise, and responsiveness to external changes and criticisms (and sometimes the co-optation of criticisms and/or competitors) are some of those rules. For instance, decision-making practices in PISA tend towards the usual solution in meta-organisations (Ahrne & Brunsson, 2006), with a preference for – and the encouragement of – consensual decisions.10 The continuity and expansion of enterprise depends on regular funding from national governments which do not automatically share the same financial capacity or the same political will to invest. The debate and the solutions proposed in 2008 concerning the financing of the research and development activities of PISA (a ‘combination’ of approaches – ‘voluntary financing primarily for research and developmental work that relates to new and separable outputs and collective financing primarily for research and developmental work relating to methodologies or longer-term research’) show how the solution to this tension may imply the development of PISA objects and the integration of social actors into PISA ‘at various speeds’.11 Knowledge generated within PISA, therefore, is/has to be adjustable.

The Plasticity of Knowledge

I also relate the expansion of PISA to specific qualities of the data, information and knowledge produced – what I call the plasticity of knowledge for policy. The knowledge used and generated by PISA surveys and reports is somewhat disciplined by the so-called ‘literacy theoretical framework’ and by particular assumptions, concepts and methods from the psychometric world (see Figure 3). But it also entails contributions generated by experts related to teaching and learning in science, reading and mathematics. The development of the surveys and the reports in each PISA domain – mathematics, reading and science – is diverse and depends on a greater or lesser consensus between experts, their permeability to political and cultural factors or pressures, and their alignment with OECD discourse.12 Furthermore, other sources of knowledge and actors participate, such as those who work on the relationships between the performances and the so-called contextual variables – which I identify here (in the circle on the right in Figure 3) as the streams of knowledge coming from policy evaluation and analysis, and from the ‘school effectiveness’ tradition.

Figure 3: Knowledge and knowledge-makers assembled within PISA.

Thus, the knowledge produced and disseminated within PISA rests on consensus among OECD staff, research consortia, experts and national representatives about what counts as usable knowledge. This plasticity – which allocates and relocates epistemic and methodological authority – is played out under the ‘transgressive’, ‘collective’ and ‘self-organised and self-authorising’ dispositions and practices already highlighted by studies on the contemporary making of expert knowledge (see Nowotny et al, 2001, ch. 14). It must be stressed that, from the point of view of those who steer PISA, these practices prove the advantages of the existence of a forum of supranational deliberation that accommodates experts and countries, and is independent of national parochialisms and interests.

The plasticity is also perceptible in PISA documents – for example, if we take into account the editing work done in some PISA main reports which creates a paratext that, step by step, summarises and selectively highlights the information from the body of the report; or if we consider the attention given by the editors to the creation of a readable document – for instance, by means of the relatively frequent introduction of texts that inform the reader how data may or may not be interpreted. In relation to concerns about the contextuality13 of the knowledge generated, and thus its adaptable properties, we may also say that throughout the PISA main reports there is a shift from moments of revelation (for instance, of core results of the survey, the performance of the students and the ‘profiles’ of literacy competence) to moments of ‘explanation’ (the interpretation of the results through relationships between variables such as socio-economic status or factors associated with national and/or school policies), and, finally, to spaces of ‘condensation’, where selected findings meet selected policy questions – school autonomy, accountability, privatisation, the involvement of stakeholders in management and financing, school choice and school improvement – most of them related to the move towards new modes of governing the school system (see Maroy, 2006).

The plasticity of PISA’s knowledge is also traceable in the variety of publications produced under the label.14 In each PISA cycle, there are several reports displaying ‘results and analyses’: the general report (prepared by the OECD Secretariat, with the support of several experts), the thematic report (on selected topics, made by external researchers) and the extensive reports (which result from partnerships between the OECD and other international agencies). Together with the national reports (elaborated by national teams), these documents express an amplification of PISA data through varied analysis. Other types of reports deal with the methodology implemented in

PISA. These are oriented towards audiences with diverse interests and skills: the technical reports display specific information about tests, scales, samples and quality control mechanisms, and the databases and the reports on the databases supply information that enables analyses to be carried out in line with the methodologies used for data collection and processing. Finally, some documents disseminate the basic assessments and are written for teachers, parents and pupils. There we may find examples of test tasks (and how they are graded), (a few) questions used in the surveys (and respective answers), and other questions used in the development and experimentation phases of the survey.

In sum, among these materials that are publicly disseminated, it is possible to detect different sorts of documents. Importantly, this variety of publications (and other informational products) has explicit target populations and opens up to a multiplicity of possible uses, whether in order to reproduce and/or recontextualise the data or information, or to produce ‘new’ knowledge from it.15 In its capacity to attach to and attract people (and their informational needs), the instrument is able to extend its reach. The ‘plasticity’ of the tool is a crucial quality in explaining why and how PISA is used differently by so many diverse actors – national researchers, the media, policy players, bureaucrats and even teachers (though they do not necessarily know they are using it). The ubiquity of the instrument rests on this feature, too.

Fabrications Added to PISA throughout Its Circulation

For more than 10 years, supplementary visions have been added to PISA and new knowledge, policies and politics have been assembled within it. These moves have multidirectional patterns. The following paragraphs give two examples of such recreations of meanings and remaking of processes.

In the first example, new fabrications are added at the supranational level in PISA management structures. Those who steer PISA are aware of and, moreover, try to create knowledge about the existing cultural and political gaps between, on the one hand, the values and regulatory forms inherent in PISA and, on the other, national mechanisms and discourses related to the regulation of the education sector. The fostering of external evaluations about the policy impact of PISA is an expressive indication of a concern with the ecology of the programme.16

Another pattern is the attempt to establish new connections or to reinforce connections that guarantee the resources and the legitimacy needed to ensure its durability. Good examples here are the organisation of open forums on PISA, the broadening of the research organisations involved in the consortia contracted to carry out the studies, and – in another direction, but a very significant one – the concern to establish positive relations with teachers’ unions. This awareness is perceptible in many other manifestations, from the strategy of the ‘modular’ structure of PISA surveys (according to the interest, vision and investment ability of the countries involved) to the regular observation of research developments in the area of performance, as well as attention to and sometimes co-optation of critical advances.17 In sum, the making of a satisfactory ecology for the survival and legitimation of PISA partly determines the path, structure and contents of the regulatory tool.

In the second example, it may be noted that research has already identified different kinds of responses to the PISA results and diverse effects on national policies, such as scandalisation, glorification or indifference (Steiner-Khamsi, 2003), or surprise, shock or promotion (Grek, 2009). Studies made within the KNOWandPOL (Knowledge and Policy in Education and Health Sectors) project also reveal this diversity as the tool travels through diverse social contexts (see Carvalho, 2009). In some spaces, the results are suppressed (Kiss et al, 2009); in others, they become part of a triennial ephemeral spectacle (Grek et al, 2009). In some spaces, PISA operates and is operated in ways such that the regulating state becomes the deregulated state (Mangez & Cattonar, 2009); in others, it functions as a provider of information that legitimates government policies (Afonso & Costa, 2009). In some countries, knowledge delivered by PISA is translated into alternative versions of a ‘common knowledge for the functioning of education’ (Mons & Pons, 2009); in others, PISA vocabulary and data become a kind of master narrative for domestic education policies (Berényi & Neumann, 2009). These studies also show diversity in the use of PISA in national or regional contexts (see Carvalho et al, 2009) – for example, its mobilisation for the making of analysis and argumentation based upon data or knowledge about specific educational policy issues; its use as a resource for the development of studies on national educational ‘realities’; or its use for the purposes of making or improving ‘domestic KRTs’, particularly in environments where accountability and/or quality assurance referents are already central. These differences are not separable from factors related to politics and culture – for example, knowledge traditions within the field of assessment, the strength of national involvement in international studies or organisations, the agenda of ongoing reforms, or the agency of policy brokers or entrepreneurs. Therefore, PISA objects and texts are reinterpreted, made acceptable and efficient for specific socio-cognitive contexts, and each context may exhibit different abilities and knowledge resources and circumstances with which to do so.

The point is that there is neither a direct transfer nor an automatic import, nor a convergence of policies. PISA frameworks, data and analysis may move away from, or closer to, the OECD agenda on educational affairs, just as they may be converted to or incorporated in (or even help to reshape) existing ‘discourse coalitions’ (see Wittrock & Wagner, 1990; Schriewer, 1998). PISA fabrications open and close pathways and establish possibilities for public action for old and new policies. When mobilised in different sociocultural spaces, they are reworked by and help to rewrite systems of meaning and relations of power, producing particular effects (on the notion of the ‘indigenous foreigner’, see Popkewitz, 2000, 2003).

Notwithstanding, the current PISA is commonly referred to and acknowledged as a valid and reliable resource for education policies and public action. In fact, one might say that PISA has achieved the condition of an obligatory point of passage (Callon, 1986; Lascoumes & Le Galès, 2007) for policy-making: it becomes an unavoidable (and ‘obvious’) provider of information ‘based on proof’, a tool that creates and allows the creation of new problems and imagined new tomorrows.

The reaction to the PISA 2009 report by the European Commissioner for Education, Culture, Multilingualism and Youth on the very day it was made public confirms that OECD scripts are taken seriously at the European Union level: from the imagination of ‘a society of smart and sustainable development’ to steering in a competitive space by mutual surveillance, ‘PISA is a useful tool that identifies the most recent trends in education by assessing our performance compared with that of the most successful countries worldwide’; from the identification of ‘problems’ – ‘the gap between the best performing education systems, and between the best performing schools and pupils within countries, remains too big’ – to praise for the ‘best performers’; and from the assurance that betterment is possible – ‘Member States ... have improved their scores in certain areas substantially’ – to the never-ending progress story – ‘Europe must continue to improve if it is to maintain its status as a knowledge society’.18

Towards a Close

The perception of the existence of conspicuous variations in the mobilisation of PISA products by diverse knowledge communities and in various policy contexts (whether these are nationally, regionally, locally or translocally situated) coexists with the recognition of its attractiveness. Divergent uses and effects coexist with a convergence towards the tool. This is neither a paradox nor an inconsistency of the instrument.

The expression ‘convergence towards the tool’ refers to the constitution of PISA as a takenfor-granted source for public policy actors, as well as to the enlargement and stabilisation of an organised space for knowledge and policy relationships. Through a complex of meditation and inquisitive activities, the programme not only connects interests, visions and projects (from those who make it and use it), but also constitutes a stabilised platform for the composition and recomposition of meanings and interests. Thus, the relevant achievement of the instrument is not to generate convergence on specific political choices or solutions (or to lead to convergent interpretations and uses of PISA products by other social actors), but, instead, the merging of policy makers and other public actors with the tool.

The power of the instrument lies in its proficiency in keeping actors and agencies that operate in different social worlds and at different levels (regional, national and supranational) bound by multiple activities of surveying, exchange and the production of knowledge. In other words, PISA survives because of the – convergent and divergent – engagement and participation of multiple actors in its own production, dissemination, use and consumption. The instrument performs an aggregation effect (Lascoumes & Le Galès, 2007). Controversies, disputes regarding the analysis of data, and competing thoughts on solutions for ‘education problems’ – the imagination and/or the scrutiny of educational systems, policies and practices – are always made by a commitment to the apparatus.

In conclusion, let us return to the concept of the policy instrument, and to the way it helps us to capture and understand public policies. As Lascoumes and Le Galès (2007, p. 3) state, each (public policy) instrument ‘constitutes a condensed form of knowledge about social control and ways of exercising it’, and each instrument produces its own effects ‘which structure public policy according to their own logic’. In the PISA case, I might say that the instrument is a carrier of guilt and helps to produce guilt: as a ‘norm and standard’ tool, it operates through the culpability and the responsibility that it conveys to national spaces and agencies. Positions (in a competitive space)and numbers bring ‘naming, blaming and shaming’ to the national policy spheres and actors. But PISA does not operate only through the power of guilt. The strength of the tool rests also in the hope it creates. The actant provides optimism for the possibility of reform, and it creates confidence in national policy actors – the encouragement of having, in ‘their hands’, the crucial ‘need to change’ signal. Moreover, the transnational actant brings the comfort of legitimising policy problems and solutions with the blessing of a putative universal, independent, expert knowledge. Concomitantly, PISA is nurtured by and nurtures several dicta on regulatory processes: the primacy of the rational and ‘based-upon-data’ model for the coordination and control of actions in the education sector, contrasting with ideological and/or opinion-based coordination; the ‘free acquiescence’ of decision makers to be involved and to (materially and symbolically) support mutual surveillance as an expected and effective practice; and the systematic assessment of student literacy performances as a useful and trustworthy resource for the steering of educational systems.

References

Adams, R. (2003). Response to ‘Cautions on OECD’s recent educational survey (PISA)’, Oxford Review of Education, 29(3), 377-389. http://dx.doi.org/10.1080/03054980307445

Afonso, N., & Costa, E. (2009). The influence of PISA on policy decision in Portugal: The education policies of the 17th Portuguese constitutional government, Sísifo, 10, 53-64.

Ahrne, G., & Brunsson, N. (2006). Organizing the world. In M.-L. Djelic, & K. Sahlin-Andersson (Eds.), Transnational governance (pp. 74-94). Cambridge: Cambridge University Press.

Barroso, J. (2006). O estado e a educação. In J. Barroso (Ed.), A regulação das políticas públicas de educação (pp. 43-70). Lisbon: Educa.

Berényi, E., & Neumann, E. (2009). Grappling with PISA: Reception and translation in the Hungarian policy discourse, Sísifo, 10, 41-52.

Bottani, N. (2006). La más bella del reino: El mundo de la educación en alerta con la llegada de un príncipe encantador, Revista de Educación, Special Issue, 75-90.

Broadfoot, P. (2000). Comparative education for the 21st century: Retrospect and prospect. Comparative Education, 36(3), 357-371. http://dx.doi.org/10.1080/03050060050129036

Callon, M. (1986). Some elements of a sociology of translation. In J. Law (Ed.), Power, action, and belief (pp. 196-233). London: Routledge & Kegan Paul.

Carvalho, L. M. (2009). Presentation note: The ‘n-moves’ of PISA – an introduction to PISA and educational public policies, Sísifo, 10, 3-13.

Carvalho, L. M. (2011). Look at the mirror! On the cognitive and normative features of a knowledge-policy tool. In J. Kusch (Ed.), Knowledge, differences and harmonies in the time of globalization (61-79). Cambridge: Cambridge Scholars.

Carvalho, L. M., & Costa, E. (2009). Production of OECD’s ‘Programme for International Student Assessment’: final report. Project KNOWandPOL, WP 11, March. http://www.knowandpol.eu/IMG/pdf/o31.pisa.fabrication.pdf

Carvalho, L. M., Costa, E., & Afonso, N. (2009). PISA – Fabrication, circulation and use in 6 European countries: Integrative report. Project KNOWandPOL, WP 17, August.

Costa, E. (2010). O Programme for International Student Assessment (PISA) como instrumento de regulação das políticas educativas. PhD thesis, Institute of Education, University of Lisbon.

Dale, R. (1999). Specifying globalisation effects on national policy: A focus on the mechanisms. Journal of Education Policy, 14(1), 1-17. http://dx.doi.org/10.1080/026809399286468

Delvaux, B. (2008, September). On the role of knowledge in the course of public action. In B. Delvaux, & E. Mangez, Towards a Sociology of the Knowledge–Policy Relation (pp. 3-97). Project KNOWandPOL, Literature Review, Integrative Report. http://www.knowandpol.eu/IMG/pdf/literature_sythesis_final_version_english.pdf

Freeman, R., Barroso, J., Ramsdal, H., & Zanten, A. van (2007). Orientation 3: Regulation: specifications. Project KNOWandPOL, Unpublished Document, September.

Freeman, R., & Maybin, J. (2011). Documents, practices and policy. Evidence and Policy, 7(2), 155-170. http://dx.doi.org/10.1332/174426411X579207

Grek, S. (2009). Governing by numbers: The PISA ‘effect’ in Europe. Journal of Education Policy, 24(1), 23-37. http://dx.doi.org/10.1080/02680930802412669

Grek, S., Lawn, M., & Ozga, J. (2009). PISA and the policy debate in Scotland: Policy narratives about Scottish participation in the international comparison. Sísifo, 10, 73-84.

Henry, M., Lingard, B., Rizvi, F., & Taylor, S. (2001). The OECD, globalisation and education policy. Oxford: Pergamon Press.

Hopmann, S. T. (2007). Epilogue: No child, no school, no state left behind. In S. T. Hopmann, G. Brinek, & M. Retzl (Eds.), PISA According to PISA: Does PISA keep what it promises? (pp. 363-415). Vienna: Lit Verlag.

Hugonnier, B. (2008). Les acquis des adolescents. Futuribles, 344, 47-61. http://dx.doi.org/10.1051/futur:200834447

Jacobsson, B. (2006). Regulated regulators. In M.-L. Djelic, & K. Sahlin-Anderson (Eds.), Transnational governance (pp. 205-224). Cambridge: Cambridge University Press.

Kiss, A., Fejes, I., & Biró, Z. (2009). Some aspects and edifications of the PISA assessments in Romania. Sísifo, 10, 65-72.

Lascoumes, P. (1996). Rendre gouvernable: De la ‘traduction’ au ‘transcodage’: L’analyse des processus de changement dans les réseaux d’action publique. In Centre Universitaire de Recherches Administratives et Politiques de Picardie (CURAPP), La gouvernabilité (pp. 325-338). Paris: Presses Universitaires de France.

Lascoumes, P. (2004). Traduction. In L. Boussaguet, S. Jacquot, & P. Ravinet (Eds.), Dictionnaire des politiques publiques (pp. 437-444). Paris: Presse de la Fondation Nationale des Sciences Politiques.

Lascoumes, P., & Le Galès, P. (2004). L’action publique saisie par ses instruments. In P. Lascoumes, & P. Le Galès (Eds.), Gouverner par les instruments (pp. 13-44). Paris: Presses de Science Po.

Lascoumes, P., & Le Galès, P. (2007). Introduction: Understanding public policy through its instruments – from the nature of instruments to the sociology of public policy instrumentation. Governance, 20(1), 1-21. http://dx.doi.org/10.1111/j.1468-0491.2007.00342.x

Latour, B. (1996). Joliot: A história e a física misturadas. In M. Sèrres (Ed.), Elementos para uma história das ciências (Vol. 3, pp. 131-155). Lisbon: Terramar. (Original work published 1989.)

Lawn, M. (2003). The ‘Usefulness’ of learning: The struggle over governance, meaning and the education space. Discourse, 24(3), 325-336. http://dx.doi.org/10.1080/0159630032000172515

Lawn, M. (2006). Soft governance and the learning spaces of Europe. Comparative European Politics, 4(2/3), 272-288. http://dx.doi.org/10.1057/palgrave.cep.6110081

Leuze, K., Martens, K., & Rusconi, A. (2007). New arenas of education governance: The impact of international organisations and markets on education policy making. In K. Martens, A. Rusconi, & K. Lutz (Eds.), Transformations of the state and global governance (pp. 3-15). London: Routledge.

Lindblad, S., Ozga, J., & Zambeta, E. (2002). Changing forms of educational governance in Europe. European Educational Research Journal, 1(4), 615-624. http://dx.doi.org/10.2304/eerj.2002.1.4.1

Lindblad, S., & Popkewitz, T. S. (2004). Educational restructuring: (re)thinking the problematic of reform. In S. Lindblad, & T. S. Popkewitz (Eds.), Educational restructuring: Perspectives on traveling policies. Greenwich, CT: Information Age.

Lindquist, E. A. (1990). The third community, policy inquiry, and social scientists. In S. Brooks, & A.- G. Gagnon (Eds.), Social scientists, policy, and the state (pp. 21-51). New York: Praeger.

Mahon, R., & McBride, S. (2008). Introduction. In R. Mahon, & S. McBride (Eds.), The OECD and transnational governance (pp. 3-22). Vancouver: University of British Columbia Press.

Mahon, R., & McBride, S. (2009). Standardizing and disseminating knowledge: The role of the OECD in global governance. European Political Science Review, 1(1), 83-101. http://dx.doi.org/10.1017/S1755773909000058

Mangez, E. (2008). Knowledge economy, knowledge policy and knowledge regimes. In B. Delvaux, & E. Mangez, Towards a Sociology of the Knowledge–Policy Relation (pp. 98-118). Project KNOWandPOL, Literature Review, Integrative Report, September. http://www.knowandpol.eu/IMG/pdf/literature_sythesis_final_version_english.pdf

Mangez, E., & Cattonar, B. (2009). The status of PISA in the relationship between civil society and the educational sector in French-speaking Belgium. Sísifo, 10, 15-26.

Marcusson, M. (2004) The Organization for Economic Cooperation and Development as ideational artist and arbitrator. In B. Reinalda, & B. Verbeek (Eds.), Decision making within international organisations (pp. 90-105). London: Routledge.

Maroy, C. (2006). École, régulation et marché: Une comparaison de six espaces scolaires en Europe. Paris: Presses Universitaires de France.

Mons, N., & Pons, X. (2009). The reception of PISA in France: A cognitive approach of institutional debate (2001-2008). Sísifo, 10, 27-40.

Morgan, C. (2007). The OECD Programme for International Student Assessment: Unraveling a knowledge network. Unpublished PhD thesis, Carleton University, Ottawa.

Nassehi, A. (2008). Making knowledge observable: Short considerations about the practice of ‘doing knowledge’. Project KNOWandPOL, Unpublished Document.

Noaksson, N., & Jacobsson, K. (2003). The production of ideas and expert knowledge in OECD. Stockholm: Stockholm Centre for Organizational Research.

Nóvoa, A. (1998). Histoire et comparaison: (essais sur l’éducation). Lisbon: Educa.

Nowotny, H., Scott, P., & Gibbons, R. (2001). Re-thinking science. Cambridge: Polity Press.

Organisation for Economic Cooperation and Development (n.d.) PISA – The OECD Programme for International Student Assessment. http://www.pisa.oecd.org/dataoecd/51/27/37474503.pdf

Organisation for Economic Co-operation and Development (OECD). (2001). Knowledge and skills for life: First results from PISA 2000. Paris: OECD.

Organisation for Economic Cooperation and Development (OECD). (2003). The PISA 2003 Assessment framework: Mathematics, reading, science and problem solving knowledge and skills. Paris: OECD.

Ozga, J. (2008). Governing knowledge: Research steering and research quality. European Educational Research Journal, 7(3), 261-272. http://dx.doi.org/10.2304/eerj.2008.7.3.261

Pons, X., & Zanten, A. van (2006). Knowledge circulation, regulation and governance. In KNOWandPOl Literature review, Volume I. http://www.knowandpol.eu/IMG/pdf/lr.tr.pons_vanzanten.eng.pdf

Popkewitz, T. S. (2000). Globalization/regionalization, knowledge, and the educational practices. In T. S. Popkewitz (Ed.), Educational knowledge: Changing relationships between the state, civil society, and the educational community (pp. 3-27). Albany: State University of New York Press.

Popkewitz, T. S. (2003). National imaginairies, the indigenous foreigner, and power. In J. Schriewer (Ed.), Discourse formation in comparative education (2nd ed., pp. 261-294). Frankfurt: Peter Lang.

Rinne, R., Kallo, J., & Hokka, S. (2004). Too eager to comply? OECD education policies and the finnish response. European Educational Research Journal, 3(2), 454-486. http://dx.doi.org/10.2304/eerj.2004.3.2.3

Rizvi, F., & Lingard, B. (2009). Globalizing education policy. London: Routledge.

Rutkowski, D. J. (2007). Converging us softly: How intergovernmental organisations promote neoliberal educational policy. Critical Studies in Education, 48(2), 229-247. http://dx.doi.org/10.1080/17508480701494259

Schleicher, A. (2007). Can competencies assessed by PISA be considered the fundamental school knowledge 15-year-olds Should Possess? Journal of Educational Change, 8(4), 349-357. http://dx.doi.org/10.1007/s10833-007-9042-x

Schriewer, J. (1998). Études pluridisciplinaires et reflexions philosophiques: la structuration du discours pédagogique en France et Allemagne. Paedagogica Historica, supp. ser. III, 57-84.

Schriewer, J. (2000). World-system and interrelationship-networks, in T. S. Popkewitz (Ed.), Educational Knowledge (pp. 305-342). Albany: State University of New York Press.

Star, S.-L., & Griesemer, J.R. (1999). Institutional ecology, ‘translation’ and ‘boundary objects’. In M. Biagioli (Ed.), The science studies reader (pp. 505-524). New York: Routledge. (Original work published 1989.)

Steiner-Khamsi, G. (2003). The politics of league tables. Journal of Social Science Education, 1, 1-6.

Steiner-Khamsi, G. (Ed.) (2004). The global politics of educational borrowing and lending. New York: Teachers College Press.

Turner, R. (2007). PISA – The Programme for International Student Assessment: An overview. Journal of Applied Measurement, 8(3), 237-248.

Vrignaud, P. (2006). La mesure de la littéracie dans PISA: La méthodologie est la réponse, mais quelle était la question? Revue Française de Pédagogie, 157, 27-42.

Weeres, J., & Kerchner, C. (1996). This time it’s serious: Post-industrialism and the coming change in education. In R. L. Crowson, W.L. Boyd, & H.B. Mawhinney (Eds.), The politics of education and the new institutionalism (pp. 135-151). London: Falmer Press.

Weinberg, M. (2008). Entrevista: Andreas Schleicher – medir para avançar rápido. Veja, 17, August, pp. 20-21. http://veja.abril.com.br/060808/entrevista.shtml

Wittrock, B., & Wagner, P. (1990). Social science and state developments. In S. Brooks, & A.-G. Gagnon (Eds.), Social Scientists, Policy, and the State (pp. 113-137). New York: Praeger.

Zanten, A. van, & Ball, S. J. (2000). Comparer pour comprendre: Globalisation, reinterpretations nationales et recontextualisations locales. Revue de l’Institut de Sociologie, 1(4), 112-131.

1 Texto original: Carvalho, L. M. (2012). The fabrications and travels of a knowledge-policy instrument. European Educational Research Journal, 11(2), 172-188. Reeditado em J. Ozga (ed.) Sociology of Education: Volume 2 – Politics and Policy (Routledge).

2 [1] A similar formulation may be found in the main report: ‘The assessment is informed – but not constrained – by the common dominator of national curricula. Thus, while OECD/PISA does assess students’ knowledge, it also examines their ability to reflect, and to apply their knowledge and experience to real-world issues’ (OECD, 2003, p. 9).

3 [2] These visions about the school system are echoed in diverse international organisations and have been developed in many other spaces, from policy analysis texts (see, for example, Weeres & Kerchner, 1996) to reports produced by specialists nominated in the mid 1990s by the (then) Commissioner for Education, as quoted – and analysed – by Lawn (2003, p. 331): ‘The future of Europe has to be constructed by several shifts: from objective to constructed knowledge; from an industrial to a learning society; from instruction to personal learning; from formal educational institutions towards new organizational structures for learning (yet to be determined) (European Commission, 1997:7)’.

4 Since 2009, the Consortium has had two nuclei: one cluster of research centres operates under the coordination of the Australian Council for Educational Research, the other under the coordination of the Dutch National Institute for Educational Measurement.

5 I will return to and develop this topic at the beginning of the following section.

6 By ecology – following Everett Hughes’ notion of the ‘ecology of institutions’ (cited in Star & Griesemer, 1999, p. 506) – I mean not only the existing contextual factors, but also the OECD’s/PISA’s choices about its sources (material, informational and human), as well as the agency skills involved in establishing continued and lasting exchanges with selected actors.

7 For instance, when considering the composition of the various expert groups, 22 persons – connected to multiple research centres – stand out, who, in the majority (20) of cases, have been linked to PISA since the late 1990s, and who, in the case of around half of them (13), make more than one type of contribution at different stages of the project and are active in different groups (for details, see Carvalho with Costa, 2009, pp. 71-72).

8 ‘We did a book – DeSeCo (Definition and Selection of Competencies: Theoretical and Conceptual Foundations) – it’s a definition and selection of competences ... that’s the source of PISA. It is a project we had over 5 years where we had anthropologists, psychologists, labor economists and all these people telling us what is competence, what does it mean to do well in life, what is a successful life and they come up with different answers ... the governments tell us that it was a waste of money because it was not conclusive and it was very sort of abstract but ... it was a really great inspiration because if we had only used educators we’d have quite the common dominator of national curriculum’ (Carvalho with Costa, 2009, p. 75; interview with an OECD executive).

9 On the role of documents in the social making of policies, see Freeman and Maybin (2011).

10 Meta-organisations operate in an institutional context that presupposes a high degree of sovereignty of its members, a principle of equality among them and, often, the goal of developing ‘similarity rather than difference among members’, although this equality is likely to be brought into question by those who invest more resources into the meta-organisation (Ahrne & Brunsson, 2006, pp. 90-92).

11 See ‘Summaries of PISA 25th Meeting of the Governing Board, 6-9 April 2008, Warsaw (Financing of Risks and Research and Development Work for PISA)’. http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=EDU/PISA/GB(2008)14&docLanguage=En

12 For a descriptive and analytical view of how the ‘evidence’ about the competences of students’ literacy is the result of agreements (which are to a greater or lesser degree precarious) within specialised scientific universes or between these and particular linguistic, cultural and political worlds, see Carvalho with Costa (2009, pp. 99-104).

13 As mentioned earlier, I use the word ‘contextuality’, following Lindquist (1990, pp. 31-35), as meaning the quality or condition of being relevant and able to be handled by an audience.

14 For more detailed information on elements of the documents produced within the scope of PISA, see Carvalho with Costa (2009, pp. 67-70) and Costa (2010).

15 The concern with the target audience of these documents is explicit: ‘For each publication the authors need to know the target population and need to be able to tailor the report with this in mind. They need to understand the contexts in which their documents will be read and used ... One of the goals of the editing process will be keeping the reports readable and the language accessible to the intended audience ... The reports will be designed to ensure that they are attractive and will match existing OECD and PISA publications to be part of a recognisable series of publications emanating from PISA’. See ‘Summaries of PISA 24th Meeting of the Governing Board, 29-31 October 2007, Edinburgh (The Process for the Development of PISA 2006 Analyses and Reports)’. http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=EDU/PISA/GB(2007)34&docLanguage=En

16 See Progress with the External Evaluation of PISA, 25th Meeting of the Governing Board, 6-9 April 2008. http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=EDU/PISA/GB(2008)10&docLanguage=En

17 For descriptions based on interviews made at the OECD level, see Carvalho with Costa (2009, pp. 73-74). [17] See the press release ‘Commission Welcomes Findings of International Student Survey (PISA) – European Results Improve’ on 8 December 2010 at http://europa.eu/rapid/pressReleasesAction.do?reference=IP/10/1680&format=HTML&aged=0&language=EN&guiLanguage=en

18 See the press release ‘Commission Welcomes Findings of International Student Survey (PISA) – European Results Improve’ on 8 December 2010 at http://europa.eu/rapid/pressReleasesAction.do?reference=IP/10/1680&format=HTML&aged=0&language=EN&guiLanguage=e

Governando a educação pelo espelho do perito 1

Luís Miguel Carvalho

Lançado em finais dos anos noventa, o Programme for International Student Assessment (PISA) é um dispositivo de avaliação comparada internacional das performances dos escolares que se vem afirmando, ao longo da presente década, como um dos principais meios de acção da OCDE no campo educativo. A agência apresenta-o como um estudo que pretende responder às exigências dos países membros no sentido de, com regularidade, disporem de dados fiáveis sobre os conhecimentos e as competências dos seus alunos e, consequentemente, sobre o desempenho dos seus sistemas de ensino. Este artigo interpela o PISA na sua condição de instrumento, baseado e gerador de conhecimento, que se propõe apoiar e participar no labor de coordenação da acção pública em educação.

A primeira secção apresenta o ponto de vista a partir do qual se aborda o PISA como instrumento de regulação. O texto parte de uma primeira interpelação do Programa, por via da sua contextualização (um breve excurso histórico sobre da tradição da avaliação comparada das performances dos escolares e sobre a intervenção da OCDE no campo educacional), para depois sumariar as principais ideias orientadoras da perspectiva analítica adoptada. Na segunda secção, o artigo descreve e analisa os elementos cognitivos e normativos do instrumento de regulação, focando a sua acção nos planos da definição da «realidade educacional», do estabelecimento de condutas apropriadas para o governo do sector da educação, e da constituição de conhecimento para a política.

Perspectivando o PISA como instrumento de regulação

Do conjunto de ideias recorrentemente disseminadas sobre o PISA em textos de auto-apresentação – textos que estabelecem as marcas da identidade do Programa, demarcando-o de projectos congéneres ou concorrentes -, sobressaem as que dão conta da natureza do empreendimento em torno de seus propósitos e meios: a sua orientação para a promoção da «aprendizagem política» e o seu carácter «colaborativo».

A vocação política do instrumento é apresentada ora como uma das características centrais, ora como um dos seus objectivos centrais:

«Its policy orientation, with design and reporting methods determined by the need of governments to draw policy lessons». (…) «PISA will aim to better assist countries in understanding the processes that shape quality and equity in learning outcomes within the educational, social and cultural contexts in which education systems operate» (…) «providing a stable point of reference against which to monitor the evolution of education systems» (OCDE, s.d., p. 1, 17).

A ligação entre o propósito da monitorização e a disposição dos agentes políticos para aprenderem com os dados é sublinhada:

“El propósito de PISA va mucho más allá de la simple supervisión del estado actual del aprendizaje por parte de los alumnos en los sistemas educativos nacionales. La información facilitada por PISA debía permitir a los responsables políticos observar qué factores están asociados al éxito educativo, y no limitarse a establecer comparaciones entre resultados aisladamente” (Schleicher, 2006, p. 23).

É igualmente sublinhado o seu carácter «colaborativo», reunindo disciplinadamente políticos e peritos, é também habitualmente realçado:

«PISA is a collaborative effort, bringing together scientific expertise from the participating countries, steered jointly by their governments on the basis of shared, policy-driven interests. Participating countries take responsibility for the project at the policy level through a Board of Participating Countries. Experts from participating countries serve on working groups that are charged with linking the PISA policy objectives with the best available substantive and technical expertise in the field of international comparative assessment of educational outcomes. Through participating in these expert groups, countries ensure that the PISA assessment instruments are internationally valid and take into account the cultural and curricular contexts of OECD Member countries, that they provide a realistic basis for measurement, and that they place an emphasis on authenticity and educational validity» (OECD, 2001, p.3).

De acordo com estes textos, a mobilização do conhecimento e dos seus peritos é feita por referência quer às coordenadas estabelecidas pelos países membros, quer ao propósito de facilitar o acesso dos políticos a saberes relevantes. Por isso, o desenho do programa e os métodos subjacentes à feitura dos relatórios finais de cada estudo são apresentados, pelos responsáveis pelo Programa, como dependentes da necessidade dos governos extraírem daí lições políticas acerca da qualidade dos resultados da aprendizagem, da igualdade nos resultados da aprendizagem e equidade nas oportunidades educativas, da eficácia e da eficiência dos processos educativos e do impacto dos resultados da aprendizagem no bem-estar social e económico (OCDE, s.d., p. 7, Schleicher, 2006, p. 23, 31).

Para compreendermos os elementos centrais da caracterização do Programa, avançados nos parágrafos anteriores – indicadores de performance, monitorização, orientação para a aprendizagem política, colaboração e parceria a uma escala global – é necessário colocá-lo em perspectiva histórica, dando atenção à sua inscrição na trajectória de instrumentos similares e ao contexto organizacional no qual é gerado.

O PISA em contexto

O PISA inscreve-se numa linhagem de estudos internacionais de avaliação comparada do rendimento dos escolares, cujo ponto de partida alguns autores localizam na década de cinquenta e associam ao trabalho conjunto de diversas instituições de investigação sob os auspícios da UNESCO (Bottani, 2006, Morgan, 2007). Desse esforço resultou um primeiro estudo (ver Foshay, 1962) e, porventura mais importante do que isso, dele dimanou, em 1961, uma forma organizacional - a International Association for the Evaluation of Educational Achievement (IEA) – que, durante cerca de três décadas e com os seus estudos (em diversas áreas, mas com mais regularidade nas da matemática, da leitura, e das ciências), viria a ocupar um lugar central – se não mesmo a obter o monopólio - no campo dos estudos internacionais, sobre as performances dos escolares (ver Husen & Postlethwaite, 1996, Postlethwaite, 1999, Bottani, 2006, Mons, 2007, Morgan, 2007, Carvalho, 2009).

A entrada em cena da OCDE, através do PISA, veio a dar término à centralidade da IEA, introduzindo rupturas numa «comunidade de prática» ao seu redor estabelecida (ver Morgan, 2007);2 mas não veio, pode dizer-se, pôr fim a uma tradição. O PISA dá continuidade à opção pela medição empírica e directa das aprendizagens como o meio para a avaliação da eficácia dos sistemas educativos (Bottani, 2006, Mons, 2007). Mas além disso, confirma um projecto de comparação (em Educação) no qual predomina uma razão pragmática, caracterizável – seguindo o modelo analítico proposto por António Nóvoa (1998) - pelo comprometimento com a construção de indicadores e standards para a decisão política, pela perspectivação da comparação como «momento da tomada de decisão», bem como, pela «crença na educação como factor de modernização/desenvolvimento».

Contudo, o Programa dispõe de características que o distinguem claramente de outros instrumentos congéneres (ver Bottani, 2006, Morgan, 2007). Umas remetem para a natureza da sua oferta, como por exemplo, a «garantia da periodicidade» (trienal), ou a «flexibilidade» da sua aplicação, em função de uma estrutura modular. De facto, a regularidade das medições e da devolução dos dados gerados – aliada à amplitude da cobertura geográfica (política e cultural) conseguida pelo Programa - constitui um dos principais trunfos exibidos pelo PISA, na medida em que permite realçar a sua capacidade de monitorização dos sistemas educativos. Ao envolverem mais de 60 países, os seus promotores podem declarar hoje que avaliam mais de um milhão de estudantes, representativos de um terço da população estudantil existente em todo o mundo e cobrindo 90% da economia mundial (OCDE, s.d., Schleicher, 2006, 2007), o que lhes permite aclamar a «pertinência universal» do instrumento (ver Hugonnier, 2008, p. 50). Outras características que permitem diferenciar o PISA de instrumentos congéneres são, porém, mais substantivas. Uma – que já abordamos em parágrafos anteriores – consiste na garantia dada aos países financiadores de poderem exercer controlo sobre as prioridades do programa e a sua aplicação, organizando as actividades em torno da construção de consensos. Outra característica consiste na «originalidade» ou singularidade do seu objecto: as competências de literacia dos jovens escolarizados. Definindo «competências de literacia» como

“[…] the capacity of young adults to access, manage, integrate and evaluate information, to think imaginatively, to hypothesize and discover, and to communicate their thoughts and ideas effectively.” (Schleicher, 2007, p. 3),

o PISA propõe-se apurar em que medida os jovens escolarizados de 15 anos: as utilizam para apreenderem e interpretarem diferentes tipos de materiais escritos, com os quais serão confrontados no seu dia-a-dia (literacia da leitura); as empregam na resolução de desafios e problemas matemáticos (literacia matemática) ou na compreensão e solução de situações e desafios científicos (literacia científica). Com o recurso a provas centradas em competências de literacia e não no currículo escolar, o PISA afasta-se da tradição dos estudos internacionais vinculados a exames e/ou matérias dos programas de ensino, estabelecendo a sua área de monopólio de conhecimento em torno de um objecto singular.

A avaliação comparada não constitui uma novidade no âmbito do trabalho da OCDE. O próprio PISA emerge no contexto do projecto International Indicators of Educational Systems (INES) que a agência desenvolve desde meados da década de oitenta (ver Morgan, 2007). Porém, com o PISA, a agência «gera os seus próprios dados». Isto é, não depende de dados já criados pelos sistemas nacionais mas, ao invés, determina o quadro de questões e de orientações que a inquirição prossegue (Henry et al. 2001). Para perceber esta novidade, vale a pena revisitar, ainda que brevemente, a trajectória da acção da OCDE sobre o sector educativo.

Apesar de ser uma organização inter-governamental direccionada para uma intervenção no campo da política económica, a OCDE vem, desde a década de 50 do século passado, intervindo na esfera educativa. Neste sector, a sua acção transitou de uma intervenção centrada no desenvolvimento de instrumentos de apoio ao planeamento educacional, no contexto do investimento dos Estados na expansão dos sistemas educativos escolares, para uma outra associada à disseminação de instrumentos de apoio à monitorização da qualidade e da eficácia dos sistemas educativos (Morgan, 2007). O que parece estar hoje em causa é o governo da educação enquanto factor gerador de vantagens na competição global, e a capacidade dos sistemas educativos produzirem uma força de trabalho «flexível», capaz de responder eficazmente às necessidades do mercado de trabalho. E é a partir de uma visão que equaciona o campo educacional como parte de uma «sociedade baseada no conhecimento» que a OCDE se propõe – com o PISA - diagnosticar e promover o desenvolvimento de competências dos escolares, pensando na sua «plena inserção» nesse espaço social imaginado.

Situando o PISA no âmbito da intervenção da OCDE no sector educativo, pode-se, portanto, fixar esta primeira ideia central: o empreendimento desenvolve-se no contexto de uma nova política educativa desta agência, marcada pela bandeira da «monitorização da qualidade» (Rinne, Kallo & Hokka, 2004). Para tal, vem a OCDE recorrendo a ferramentas diversas - estatísticas, comparações internacionais, identificação de boas práticas e de boas políticas. Desenvolver esses instrumentos, vocacionados para «ajudar a decisão política», constitui aliás, um papel assumido pela organização, como lembra o seu Director-adjunto do Departamento de Educação:

«[PISA] «(…) développé par l’OCDE répond à cette exigence. L’objectif the nombreux pays est désormais d’améliorer la gouvernance publique et d’introduire peu à peu une culture d’évaluation. Être à même de comparer les investissements avec les résultats est donc indispensable. En mesurent les performances des élèves, PISA rend possible cette comparaison. Par ailleurs, en apportant nombre d’éclairages sur les pratiques en vigueur dans les pays participants, PISA offre la possibilité de mieux comprendre la nature des problèmes et d’identifier des réformes possibles» (Hugonnier, 2008, p. 48).

Uma outra ideia central a reter acerca da acção da OCDE no campo educacional respeita ao modo como exerce o seu papel de actor político independente (Henry et. al, 2001). A OCDE difere de outras organizações que operam a uma grande escala por não recorrer a instrumentos legais nem financeiros e por agir, sobretudo, através da «construção de consensos» e da «pressão pelos pares» (Rinne, Kallo & Hokka, 2004, p. 455-6). A capacidade da OCDE para desenvolver e dar forma às decisões políticas revela-se segundo duas modalidades: através de uma governança pela «coordenação», fazendo convergir actores diversos em iniciativas comuns, como conferências ou projectos; através da «formação de opinião», mediante um trabalho de produção de visões e de valores, de modelos e conceitos, que lhe permite iniciar e influenciar os discursos nacionais em matéria educacional (Leuze, Martens & Rusconi, 2007).

Em suma, em vez de comandos ou de directivas, predominam as práticas de uma regulação política soft, onde sobressaem a construção de regras, a monitorização e a construção de agendas para as políticas (Jacobson & Sahlin-Anderson, 2006, Mahon & McBride, 2008). Estas práticas manifestam-se em actividades «meditativas» e «inquisitivas». No primeiro caso, as estruturas e processos de intervenção dos Estados modernos são debatidos por experts, e aí se fixam standards e se propõem modelos inovadores. No segundo caso, os Estados envolvidos dispõem-se a mostrar-se aos outros e a submeterem-se a examinação, através de auditorias, avaliações, rankings (Jacobson, 2006, pp. 207-208).

O PISA é um bom exemplo dessa intervenção da OCDE. Combinando actividades de «inquisição» e de «meditação», o Programa é palco da promoção de acordos, acerca de quais são as práticas e as políticas educacionais que os governos nacionais «admitem submeter a escrutínio externo», e municia regularmente os políticos de dados e análises construídos a partir de modelos construídos a partir de convenções estabelecidas entre peritos.

Estudando o PISA como instrumento de regulação

Encontramos no PISA as características de um «instrumento de acção pública» (Lascoumes & Le Galès, 2004) que reúne e entrelaça componentes técnicas (medida, cálculo, procedimento) e sociais (representações, símbolos) e que tanto é portador de representações e de problematizações específicas do universo educativo, como participa na organização das relações sociais específicas entre actores, introduzindo regras, normas, procedimentos que intentam dar estabilidade e previsibilidade à acção colectiva e individual no universo educativo.

Assim, os indicadores da literacia matemática, científica, de leitura, etc., que o PISA providencia não são, nestas páginas, entendidos como sinalizadores da capacidade que os países avaliados têm para promoverem aprendizagens nos seus sistemas escolares. O que interessa a este texto são: as regras e as normas que o PISA fixa ou induz; as formas «adequadas» de prover educação que põe em equação; o exercício de mútua observação em que coloca – regularmente - os decisores políticos num espaço «competitivo-cooperativo» mundial.

Consequentemente, o presente artigo observa e analisa o Programa nessa sua qualidade de instrumento portador de modos específicos de pensar a «realidade educativa» e de regras que introduzem orientação, coordenação e controlo nas acções e nos actores do sector educativo. Portanto, o PISA é aqui analisado enquanto uma «tecnologia política» (Ozga & Grek, 2007, Grek, 2009) que toma parte na fabricação das pessoas para uma sociedade (global) imaginada, bem como na geração de formas particulares para o seu governo, assentes em dispositivos «baseados no conhecimento».

Nos estudos desenvolvidos no âmbito do projeto KNOWandPOL (Carvalho, 2009) vimos estudando os processos de fabricação do PISA no cruzamento de duas dimensões: social e cognitiva. A primeira refere-se à criação e estabilização de processos de organização, de cooperação, entre os múltiplos mundos sociais envolvidos no Programa. A segunda diz respeito às escolhas e definições acerca do que conta como conhecimento para a política (e dos modos de produção e dos conteúdos que compreende), bem como acerca de uma visão para educação e dos modos como esta pode/deve ser governada. O texto explora, nas páginas seguintes, esta segunda dimensão da análise do instrumento.

Analisando a dimensão cognitiva do instrumento de regulação

As concepções, os critérios e os indicadores pelos quais as sociedades nacionais, os sistemas educativos e os desempenhos dos estudantes são descritos, comparados e avaliados, no âmbito do PISA, operam a um nível ontológico (Alasuutari, 2005). Entendemo-los aqui como categorias discursivas, em linha com a reflexão de Tom Popkewitz (2000, p. viii):

“ (…) discursive practices whose distinctions and categories order the world, define the places of individuals, and establish rules and standards by which expression and practices are made possible”.

Debruçamo-nos sobre esses elementos cognitivos e normativos do PISA, através de três «golpes» analíticos, considerando: a definição da «realidade educacional»; a definição das condutas apropriadas para o governo do sector da educação; e a construção de conhecimento para a política.

Redefinindo categorias da escolarização

O PISA é um dispositivo de monitorização comparada que delineia uma forma de olhar e de agir na educação. Essa acção exerce-se sobre categorias particulares da escolarização senão mesmo sobre todo o aparato escolar.

Convém recordar que o PISA traz um novo enfoque para as avaliações internacionais:

«rather than examine mastery of specific school curricula, PISA looks at students’ ability to apply knowledge and skills in key subject areas and to analyse, reason and communicate effectively as they examine, interpret and solve problems». (PISA Website)

Este enfoque provoca um deslocamento da inquirição para fora do espaço estrito das disciplinas escolares e dos saberes dos programas escolares. Este deslocamento é, em si mesmo, um duplo afastamento: (i) por um lado, das escolhas nacionais sobre as matérias de ensino e sobre as aprendizagens por matéria de ensino; (ii) por outro lado, dos estudos anteriormente efectuados por outras agências (por exemplo os estudos do IEA). Este movimento permite ao projecto PISA uma dupla distinção: com a convencional reflexão dos sistema escolares sobre si mesmos; com a estabelecida conceptualização dos estudos científicos comparados sobre as performances dos alunos na dependência das categorias próprias aos sistemas educativos. Por fim, e paralela a esta diferenciação, produz-se uma reavaliação do conhecimento escolar. Como escreve Mangez (2008, pp. 102- 104), a noção de competência sanciona uma perspectiva utilitarista do conhecimento, na medida em que faz da «utilidade» (utilização para resolver problemas do dia-a-dia) o critério para a avaliação do saber escolar. Ela permite, assim, uma reestruturação da composição curricular, não nos termos convencionais – das «disciplinas» - mas em termos de cruzamento disciplinar ou transdisciplinar.

A noção de competência de literacia entrelaça-se, por outro lado, com uma perspectiva científica particular sobre a aprendizagem, cuja adopção legitima à luz das mudanças da economia. A necessidade de uma passagem de uma teoria comportamental da aprendizagem, para uma teoria cognitiva da aprendizagem, tem sido enfatizada por aqueles que reclamam a adaptação do sistema escolar às mutações da economia, do industrialismo para o pós-industrialismo3. O quadro de referência do PISA faz eco dessa orientação, como se pode ler na justificação da abordagem que é seguida no estudo das «competências da literacia científica»:

«This approach was taken to reflect the nature of the competencies valued in the modern societies, which involve many aspects of life, from success at work to active citizenship. It also reflects the reality of how globalisation and computerisation are changing societies and labour markets. (…) This analysis shows that the steepest decline in task input over the last decade has not been with manual tasks, as often reported, but with routine cognitive tasks, i.e. those mental tasks that are well describe by deductive or inductive rules, and that dominate many of today’s middle-class jobs. This highlights that if students learn merely to memorize and reproduce scientific knowledge and skills, they risk being prepared mainly for jobs that are disappearing from labour markets in many countries. In order to participate fully in today’s global economy, students need to be able to solve problems for which they are no clear rule based solutions and also communicate complex scientific ideas clearly and persuasively. PISA as responded to this by designing tasks that go beyond the simple recall of scientific knowledge» (OCDE, 2007, p. 33)

As consequências da adopção de uma perspectiva cognitivista traduzem-se em descrições e proposições acerca de mudanças nos modos de organizar o ensino-aprendizagem: do aprendiz “passivo” e “encorajado a ser passivo”, ao aprendente “experimentador e empenhado”; do professor que informa e corrige, no interior de uma “estrutura hierárquica de aprendizagem”, ao professor que recorre a modelos orgânicos (como o branching e o networking) e que “devolve autoridade e responsabilidade” ao aluno; da exterioridade do conhecido (corpos de conhecimento sequencialmente transmitidos), à interioridade da construção do conhecimento em que a aprendizagem começa com o estabelecimento pelo aluno de uma “conexão com a aquilo que já sabe” sobre o que vai estudar (acompanhando Weers & Kerchner, 1992, p. 147).

Estes deslocamentos discursivos não atingem apenas categorias particulares da escolarização. Implicam a própria escolarização no seu conjunto. Conectando directamente a aprendizagem, os contextos e os modos de ensinar/aprender, com as «exigências» do ambiente do sistema escolar, os indicadores do PISA firmam o olhar e o debate sobre os efeitos dos sistemas escolares, não ao redor dos resultados, mas em torno das suas consequências (sobre outros sistemas). A «caixa negra» que pretendem abrir é a instituição escolar, cujo fechamento se manifesta, argumentam, na perpetuação de práticas de (auto) descrição e de (auto) avaliação que apenas consideram os seus inputs e/ou os seus outputs (resultados de testes e de exames a partir dos objectivos e conteúdos das matérias de ensino). A «visão PISA» é a de que a escolarização não deverá, portanto, ter o seu produto avaliado por referência a si mesma e às suas categorias convencionais. Antes deve pensar-se em função das consequências do que faz, prestando assim contas (aos outros sistemas) daquilo que faz.

Ao introduzir um questionamento de práticas e cerimoniais avaliativos da instituição escolar ou das modalidades «tradicionais» de organizar o ensino-aprendizagem, o PISA pode ser percebido, por muitos actores sociais, enquanto portador de um potencial de mudança, como força crítica e de des-institucionalização. Mas ele constitui – simultaneamente - uma proposta de reescrita do modelo escolar, uma semântica que dirige a sua adaptação às lógicas da sociedade e da economia do conhecimento (e respondente a estas), ou mesmo, de uma mais ampla reescrita do modelo educativo das sociedades contemporâneas.

Os decisores políticos como aprendizes e a comparação como forma de governo

O PISA, em suas modalidades de organização, de inquirição e de publicação, incorpora e cria espaços para as políticas da cognição, ou seja para a agência sobre as visões do mundo, as percepções da realidade e as preferências. Essa intervenção do PISA não se esgota na problematização da «realidade educativa». Exerce-se também no campo da reordenação da acção dos seus potenciais utilizadores – em particular, dos decisores políticos nacionais.

O universo discursivo do PISA incorpora e expressa modos de pensar e de fazer interiorizáveis pelos actores políticos, para que eles se percebam capacitados para participar “naturalmente” nos exercícios de monitorização e de comparação internacional, tomados estes como práticas correntes e apropriadas às suas identidades e papéis. Neste quadro, o PISA pressupõe e define, também, um certo tipo de decisor político e um certo tipo de racionalidade política.

Em elocuções que descrevem e avaliam as relações dos políticos com o próprio PISA, podemos encontrar essas regras que qualificam/desqualificam o decisor político. Seguimos, para tal, uma entrevista concedida por Andreas Schleicher à revista Veja, editada sob o sugestivo título «Medir para Avançar Rápido»:

«(…) o Brasil passou a ter chance de avançar no momento em que começou a mapear os problemas de maneira objectiva – e não mais com base na intuição de alguns governantes. Isso é básico. Não dá sequer para melhorar algo que não foi sequer dimensionado. Daí a importância da comparação internacional. Ao olhar os rankings, pais, educadores e autoridades podem começar a fazer comparações e constatar o óbvio: suas escolas estão bem atrás das dos países da OCDE.» (…) «Ao saberem do fiasco nos últimos rankings, alguns políticos e especialistas de mentalidade mais atrasada me ligaram revoltados. Diziam: ‘Vocês estão exigindo dos alunos que falem sobre situações distantes demais da realidade deles. É injusto’. A miopia dessa gente impede de enxergar que o fato de estudantes chineses ou americanos terem resposta para tais questões não revela apenas o despreparo dos brasileiros, mas mostra também como eles estão em desvantagem na competição com os demais.» (…) «Apenas há pouco tempo, as autoridades passaram a usar esse medidor tão eficaz para aferir as próprias deficiências e apontar saídas com base em experiências que dão certo noutros países. (…) Os chineses não demonstram constrangimentos em copiar o que funciona noutros países. Ao contrário: eles são movidos por isso (…) A China, evidentemente, ainda tem muito que melhorar na educação – mas avança a um ritmo veloz» (Schleicher, 2008, pp. 20-21).

Do ponto de vista de quem dirige o PISA – mas que para efeitos desta análise corporiza e expressa o conjunto de pressupostos que estabelecem a apropriada conduta do actor político no universo da agência OCDE – o bom decisor político é aquele que: governa diagnosticando de modo objectivo o seu mundo ou sector; se orienta pela procura de vantagens competitivas através da medição dos outcomes do sistema escolar; identifica fragilidades e adopta soluções a partir do what works em outros países; conhece e copia os competidores para poder progredir mais rapidamente. O bom político é também o que dá ao sector da educação «prioridade» política e que a pensa pela razão económica, ou seja, rege-se pelo princípio da competitividade (viu-se atrás), pondo em marcha medidas «eficazes», e gere com «eficiência» as verbas orçamentais para o sector educativo. O bom decisor político não é apenas o que gere o sistema mas sim, aquele que: muda o sistema; tira o sector educativo do «atraso» – por comparação com outros sectores tomados como equivalentes; faz o seu sector «produzir» mais e melhor; movendo-o do «modelo industrial» para o modelo «pós-industrial» (ver Weinberg, 2008).

É dentro desta teia de causas e efeitos, de condições e de consequências (presentes e futuras), que a acção dos políticos é compreendida e que os critérios para a avaliação das suas condutas são fixados:

«Nous vivons à l’heure de l’économie de la connaissance et de la mondialisation. L’une et l’autre exigent une plus haute efficience des services publics, ce qui ne peut être atteint que si l’ont peu comparer les investissements réalisés avec leurs résultats, et si les décisions politiques sont en pleine connaissance de cause. PISA répond à ces deux exigences en fournissant des informations inédites sur les performances comparées de leur système éducative. (…) Sans ces informations (…) les décisions politiques fondées davantage sur des perceptions que sur les réalités présentent de forts risques d’échec, avec les coûts considérables en termes financiers mais aussi humains que l’ont peut imaginer» (Hugonnier, 2008, p. 61).

Neste contexto argumentativo, os ‘resultados do PISA’ são apresentados como capazes de fornecer ao decisor político uma base de comparação «segura», para conhecer não apenas o lugar do ‘seu’ sistema educativo em um espaço competitivo mundial, mas também o seu lugar numa linha do tempo (que vai da sociedade industrial à sociedade do conhecimento). Com os ‘dados PISA’ e os modos de pensar PISA, os decisores políticos podem orientar os seus próprios movimentos e podem fazer mover o seu país para o ‘tempo’ (físico e simbólico) do «mundo de amanhã». Com, ou através do PISA, o decisor politico racional é aquele que governa com/através de uma razão comparada – e a mútua vigilância é a regra para o bom governo (ver Nóvoa & Tariv-Mashal, 2003).

A montagem e a flexibilidade do «conhecimento para a política»

Os documentos difundidos no âmbito do PISA – mormente aqueles que divulgam dados e análises - são preparados sob certas convenções acerca de «o que é conhecimento para a política». Eles podem ter vários propósitos, como por exemplo: permitir que os governos possam monitorizar os seus sistemas educativos, apoiados em medições consistentes e confiáveis das performances dos estudantes; ou gerar conhecimento acerca de «questões decisivas» em matéria de sucesso educativo - e dos seus meios de criação e difusão desse conhecimento – através de «um esforço colaborativo» que reúne experts e políticos. Quando analisamos as afirmações contidas nesses documentos acerca destes propósitos, percebemos que a sua lógica vai para além dos pressupostos da tradicional visão «racionalista» acerca das relações entre conhecimento e política.4 Apesar de incorporados nos textos PISA, essas regras não são os ingredientes únicos. Mais ainda, são reconfiguradas no interior de uma diferente lógica. Salientamos, seguidamente, três elementos de transfiguração.

Primeiro: o ponto de partida dos inquéritos PISA é parcialmente definido por outros que não os investigadores e o trabalho desenvolvido ao longo da trajectória de cada ciclo do Programa pode ser alvo de escrutínio pelos políticos. Segundo: a produção de conhecimento envolve vários grupos de investigadores, especializados em áreas diversas, que colectivamente validam a sua expertise. Terceiro: os resultados das pesquisas não são trabalhados de modo a oferecer a solução do conhecimento ao problema da política, mas de modo a facilitar uma relação de aprendizagem entre a audiência e o conhecimento disponibilizado. Dito de outro modo, os dados e as análises são preparados de modo a serem usados pelo decisor político, ao qual se atribui a responsabilidade de contemplar, de analisar, de reflectir e, daí, de extrair lições para a política pública. Portanto, não se trata de aproximar «linguagens», mas já de engajar os políticos em relações específicas de criação de conhecimento a partir do conhecimento disponibilizado.

Associamos estes três elementos à mutação das «regras do jogo» da fabricação de conhecimento para a política. Mudança já retratada em literatura dedicada à análise das narrativas dos peritos contemporâneos (Nowotny, Scott & Gibbons, 2001) e que sublinha a disposição destes para a produção de um conhecimento «transgressivo», «colectivo», fundado na «auto-organização» e na «auto-autorização». Mas associamos também esses elementos a imperativos de acção da própria OCDE, cujo sucesso não depende só das actividades de produção e de difusão de conhecimento. Requer, também, a manifestação das capacidades de agir e de ser reconhecido como gerador de um conhecimento que deve ser ao mesmo tempo: credível em universos científicos, mas manuseável pelas audiências políticas; percebido como portador de uma visão inovadora e independente da realidade educacional, mas assente em consensos.

Este conjunto de características e tensões é observável quer nos processos de inquirição quer nos principais relatórios publicados (OCDE, 1999, 2001, 2003, 2004, 2006, 2007). Estas tensões revelam-se naquilo que designamos por processos de «montagem» (assemblage) do conhecimento e pelo carácter «flexível» - ou pela plasticidade - do conhecimento PISA (Carvalho, 2009, p. 95-110).

Eis alguns exemplos.

O Programa cria e difunde, primariamente, conhecimento sobre as performances e o empenhamento dos estudantes, sob o enquadramento conceptual das «competências de literacia». É isto que diferencia o «conhecimento PISA» daquele que é gerado em empreendimentos que com ele podem competir. A elaboração deste conhecimento depende de um conjunto de fontes e de produtores (e.g., conhecimento proveniente de estudos/experts na leitura, na matemática, no ensino das ciências, proveniente da estatística, da psicometria e da avaliação comparada). Embora seja «disciplinada» pelo quadro conceptual das «competências de literacia» e pela validação da comparabilidade construída no mundo da psicometria, a literacia é, porém, construída diferentemente em cada um dos domínios (leitura, matemática, ciências). Essa variabilidade depende de vários factores: do maior ou menor consenso obtido no interior de cada grupo de especialistas; da sua permeabilidade a factores e pressões político-culturais (nomeadamente introduzidas pelos representantes dos países membros no órgão de direcção do PISA); da sua maior ou menor afinidade com as categorias discursivas empregues pela OCDE na legitimação da universalidade do seu projecto de avaliação comparada das performances, por exemplo, com o racional da economia global e da sociedade do conhecimento.

Nos Relatórios Finais publicados pela OCDE no final de cada ciclo PISA, as «evidências» sobre as performances dos alunos (material primário e singular do PISA) são trabalhadas em relação com variáveis contextuais. Esta é uma segunda área de «conhecimento» sobre a realidade educativa, para a qual outros ramos de saberes são mobilizados (análise e avaliação de políticas e estudos sobre a eficácia das escolas) e na qual o saber singular do PISA é reunido com factores associados, pelos analistas, às estruturas dos sistemas educativos e às escolhas dos políticos. Várias mudanças têm ocorrido neste espaço de conhecimento. Destacamos duas. (1) A criação de um território analítico especificamente associado a elementos centrais do discurso actual da OCDE – o par «equidade/qualidade” – e no qual a oposição entre «eficiência social» e «equidade social» é transformada em matéria de «prova» técnico-científica. (2) A importância adquirida por um conjunto de factores cujos conteúdos retomam temas centrais das mudanças da regulação das políticas públicas nas últimas décadas: autonomia das escolas, accountability, privatização, livre escolha da escola, melhoria da escola….

Podemos observar que no interior de cada uma destas publicações, a par de um saber usado para desenvolver o survey PISA, emerge outro conhecimento gerado pelos actores que trabalham os resultados dos inquéritos. Os relatórios incorporam o primeiro, mas também lhe dão uma outra magnitude e natureza. Nos relatórios, o trabalho que prepara os surveys – e.g., preparação dos quadros conceptuais, construção da instrumentação de recolha ou de análise estatística – são transformados em «resultados», em «análises» e em «implicações». Noutros termos, os relatórios mobilizam e fixam conhecimento gerado ao longo da inquirição, mas prolongam-no e associam-no a novos sentidos. O conhecimento PISA é aproximado da audiência, movendo-se da revelação para a condensação. Os materiais centrais e originais do PISA (as performances dos estudantes e os perfis de competência de literacia de estudantes e países) dão lugar, seguidamente, à análise da sua variabilidade em função de relações estabelecidas com factores de contexto (e.g., estatuto socioeconómico dos estudantes) e com factores dependentes da decisão política nacional. Depois terminam em conclusões-chave e com a identificação de «factores» que devem ser tomados em consideração como questões políticas cruciais. Tudo isto feito não sob a forma de um texto prescritivo, antes redigido como um script destinado à análise, à avaliação e à reflexão dos políticos.

Sublinhe-se pois, para finalizar, o carácter estruturado e estruturável do conhecimento mobilizado para o PISA e gerado no seu interior. Trata-se de uma montagem - resultante de contingências e de estratégias - onde estão inscritas ideias e actores sociais diversos, como por exemplo: a perspectiva pragmática dos estudos comparados; a tradição da ciência positiva; os sofisticados métodos quantitativos da psicometria; os consensos pré-existentes ou gerados no interior das comunidades e entre as comunidades de saberes especializados em matemática, leitura e ciências; o universo dos discursos sobre as sociedades contemporâneas; o estabelecimento de consensos entre os actores políticos acerca do que admitem ver/ter (mutuamente) escrutinado, e entre estes e todos os experts. Esta montagem e as relações que a suportam fazem pensar nas ‘competências de literacia’ como um objecto sempre actualizável. Não obstante, elas se afirmam (e com elas o PISA) como um ponto de origem e de chegada das políticas: um ponto a partir do qual as políticas se podem/devem orientar e para onde se podem/devem orientar.

Fecho: A mútua vigilância pelo espelho do perito

« (…) we are more like a mirror. We have a mirror; you can look at it and you can see how the world looks; if you don’t like it you don’t look at it».5

Os relatórios do PISA são apresentados pelos seus promotores – ou querem ser percebidos - como um espelho que «apenas» revela, com limpidez e nitidez científica, «a realidade» das performances dos escolares e de suas relações com os seus backgrounds sócio-económicos e com os factores de mediação (de intervenção política). Para quem dirige o Programa, o que mais importa é que este seja reconhecido como ponto de passagem obrigatório para um debate e uma decisão política centrados em evidências científicas – criando ‘novos’ problemas e disponibilizando novas informações ‘baseadas em evidências’ – e, não tanto, ser uma instância que resolve problemas (ver Carvalho, 2009). Os surveys e os relatórios PISA são parte fundamental da exibição – pela OCDE - de uma capacidade constante de produção de inovação nos discursos políticos nacionais, de introdução de rupturas com os estereótipos da acção pública e de diferenciação face às ideias políticas nacionais.

Pelo que argumentamos em momentos anteriores deste texto, o «espelho do perito» traz consigo muito mais que uma simples devolução da realidade. A realidade devolvida pelo espelho compreende, articula e difunde regras acerca da educação, da escolarização, e das relações entre estas e a economia e a sociedade, bem como, mobiliza, estrutura e divulga modelos para a compreensão dessas «realidades». Um dos elementos centrais desta intervenção sobre as categorias de percepção reside no espelho que mostra cada um (cada país) na sua relação e na sua posição relativa a outros. Como já foi referido atrás, a comparação situa os decisores políticos numa imaginada linha do tempo que vai da sociedade industrial à sociedade do conhecimento e que se desenrola num espaço competitivo-cooperativo mundial. A comparação internacional é um lugar onde se exerce a imaginação de uma sociedade transnacional (traçando para os indivíduos um quadro comum de expectativas de performance e de empenhamento na vida social a partir da confiança depositada na expertise científica) e no qual se constrói uma polity transnacional (na qual os Estados nacionais voluntariamente se envolvem com outras instâncias na produção conjunta de regras). Assim, a agência da OCDE não produz apenas sentidos e conteúdos para as políticas nacionais, não actua apenas como fonte de regulação dessas políticas «nacionais». Ela produz também um conjunto de regras para a regulação política dos reguladores das políticas.

Referências

Alasuutari, P. (2005, July). The governmentality of consultancy and cooperation: The influence of OECD. Paper presented at the 37th World Congress of the International Institute of Sociology. Stockholm.

Barroso, J., & Carvalho, L. M. (2009). PISA: Un instrument de régulation pour relier des mondes. Revue Française de Pédagogie, 164, 177-180.

Bottani, N. (2006). La más bella del reino: El mundo de la educación en alerta con la llegada de un príncipe encantador. Revista de Educación, extraordinario, 75-90.

Broadfoot, P. (2000). Comparative education for the 21st century: Retrospect and prospect. Comparative Education, 36(3), 357-371.

Carvalho, L. M. [com Estela Costa]. (2008, April). Production of OECD’s ‘Programme for International Student Assessment: KNOWandPOL Orientation 3 – Supra-national Instruments – WP 11 Report (Education). Disponível em: http://www.knowandpol.eu/

Delvaux, B., & Mangez, E. (Eds.). (2007, June). Literature reviews on knowledge and policy: KNOWandPOl Literature Review - Report 1. Disponível em: http://www.knowandpol.eu/

Delvaux, B., & Mangez, E. (2008, September). Towards a sociology of the knowledge-policy relation: KNOWandPOL literature review – integrative report. Disponível em: http://www.knowandpol.eu/

Foshay, A. W. (Ed.). (1962). Educational achievements of thirteen–year-olds in twelve countries. Hamburg: UNESCO Institute for Education.

Gagnon, A.-G. (1990). The influence of social scientists on public policy. In S. Brooks, & A.-G. Gagnon, (Eds.), Social Scientists, Policy, and the State (pp.1-17). New York. Praeger.

Grek, S. (2009). Governing by numbers: The PISA effect. Journal of Education Policy, 24(1), 23-37.

Henry, M., Lingard, B, Rizvi, F., & Taylor, S. (2001). The OECD, globalization and education policy. Oxford: Pergamon Press.

Husén, T., & Postlethwaite, T. N. (1996). A brief history of the International Association for the Evaluation of Educational Achievement (IEA). Assessment in Education, 3, 129-141.

Huggonnier, B. (2008). Les acquis des adolescents. Futuribles, 344, 47-61.

Jacobson, B. (2006). Regulated regulators. In M.-L. Djelic, & K. Sahlin-Anderson (Eds.), Transnacional governance: Institutional dynamics of regulation (pp. 205-224). Cambridge: Cambridge University Press.

Jacobson, B., & Sahlin-Anderson, K. (2006). Dynamics of soft regulations. In M.-L. Djelic, & K. Sahlin-Anderson (Eds.), Transnacional governance: Institutional dynamics of regulation (pp. 247-266). Cambridge: Cambridge University Press.

Lascoumes, P., & Le Gales, P. (2004). L’action publique saisie par ses instruments. In P. Lascoumes, & P. Le Gales (Dir.), Gouverner par les instruments (pp. 13-44). Paris: Presses de Science Po.

Latour, B. (1996). Joliot: A história e a física misturadas. In M. Sèrres (Dir.), Elementos para uma história das ciências (III volume, pp. 131-155), [orig. 1989]). Lisboa: Terramar.

Leuze, K., Martens, K., & Rusconi, A. (2007). New arenas of education governance – the impact of international organizations and markets on education policy making. In K. Martens, A. Rusconi, & K. Lutz (Eds.), Transformations of the state and global governance (pp. 3-15). London: Routledge.

Lindblad, S., & Popkewitz, T. S. (2005). Educational restructuring: (re)thinking the problematic of reform». In S. Lindblad, & T. S. Popkewitz (Eds.), Educational restructuring: Perspetives on traveling policies. S.l.: IAP.

Lindquist, E.A. (1990). The third community, policy inquiry, and social scientists». In S. Brooks & A.-G. Gagnon (Eds.), Social Scientists, Policy, and the State (pp. 21-51). New York. Praeger.

Mahon, R., & Mcbride, S. (2008). The OECd and transnational governance. Vancouver: UBC Press.

Mangez, E., (2008, September). Knowledge economy, knowledge policy and knowledge regimes In B. Delvaux, & E. Mangez, Towards a sociology of the knowledge-policy relation: KNOWandPOL literature review – integrative report (pp. 99-118). Disponível em: http://www.knowandpol.eu/

Mons, N. (2007). L’évaluation des politiques éducatives. Apports, limites et nécessaire renouvellement des enquêtes internationales sur les acquis des élèves. Revue Internationale de Politique Comparée, 14(3), 409-423.

Morgan, C. (2007). The OECD Programme for International Student Assessment: unraveling a knowledge network. Thesis (Doctoral). School of Public Policy and Administration. Carleton University, Ottawa.

Nóvoa, A. (2002). Ways of thinking about education in Europe. In A. Nóvoa, & M. Lawn (Eds.), Fabricating Europe: The formation of an education space (pp. 131-155). Dordrecht: Kluwer Academic Publishers.

Nóvoa, A., & Tariv-Mashal, T. (2003). Vers un Comparatisme Critique. Lisboa: Educa.

Noaksson, N., & Jacobsson, K. (2003). The production of ideas and expert knowledge in OECD. The OECD jobs strategy in contrast with the EU employment strategy. Stocholm: SCORE.

OCDE (s. d.). PISA – The OECD Programme for International Student Assessment. [Brochura PISA]. Disponível em http://www.pisa.oecd.org/dataoecd/51/27/37474503.pdf

OCDE (1999). Measuring student knowledge and skills: a new framework for assessment. Paris: OECD.

OCDE (2001). Knowledge and skills for life: First results from PISA 2000. Paris: OECD.

OCDE (2003). The PISA 2003 assessment framework: Mathematics, reading, science and problem solving knowledge and skills. Paris: OECD.

OCDE (2004). Learning for tomorrows world: First results from PISA 2003. Paris: OECD.

OCDE (2006). Assessing Scientific, Reading and Mathematical Literacy: A framework for PISA 2006. Paris: OECD.

OCDE (2007). PISA 2006: Sciences competencies for tomorrows world. Paris: OECD.

Ozga, J., & Grek, S. (2007, September). Governing by numbers. Paper presented to the European Sociological Association Annual Conference, Network 10, Glasgow.

Popkewitz, T. S. (2000). Preface. In T. S. Popkewitz (Ed.), Educational knowledge: Changing relations between the state, civil society, and the educational community (pp. vii-x). New York: State University of New York Press.

Postlelthwaite, T. N. (1999). Overview of issues in international achievement studies. In B. Jaworski, & F. Phillips (Eds.), Comparing standards intenationally (pp. 23-59). Oxford: Symposium Books.

Rinne, R., Kallo, J., & Hokka, S. (2004). Too eager to comply? OECD education policies and the Finnish response. European Educational Research Journal, 3(2), 454-485.

Rizvi, F. (2004). Theorizing the global convergence of educational restructuring. In S. Lindbad, & T. S. Popkewitz (Eds.), Educational restructuring: perspetives on traveling policies (pp. 21-41). S.l.: Information Age Publishing.

Rochex, J. Y. (2006). Introduction [Dossier PISA: analyses secondaires, quetsions et débats théoriques et méthodologiques]. Revue Française de Pédagogie, 157, 5-9.

Rutkowski, D. J. (2007). ‘Converging us softly’: How intergovernmental organizations promote neoliberal educational policy. Critical Studies in Education, 48(2), 229–247.

Schleicher, A. (2006). Fundamentos y cuestiones políticas subyacentes al desarollo de PISA. Revista de Éducación, extraordinário, 21-43.

Schleicher, A. (2007). Can competencies assessed by PISA be considered the fundamental school knowledge 15-year-olds should possess? Journal of Educational Change, 8(4), 349-357.

Star, S.-L., & Griesemer, J. R. (1999). Institutional ecology, ‘translation’ and ‘boundary objects”. In M. Biagioli (Ed.), The science studies reader (pp. 505-524) [orig. 1989]. New York: Routledge.

Weinberg, M. (2008, Agosto). Entrevista: Andreas Schleicher – Medir para avançar rápido. Veja, 17, 20-21. Disponível em: http://veja.abril.com.br/060808/

Weers, J., & Kerchner, C. (1996). This time it’s serious: Post-industrialism and the coming change in education. In R. L. Crowson, W.L. Boyd, & H. B. Mawhinney (Eds.), The Politics of Education and the New Institutionalism (135-151). London: Falmer Press.

1 Texto original: Carvalho, L.M. (2009). Governando a educação pelo espelho do perito. Educação e Sociedade (Brasil), 30(109), 1007-32. Versão revista. Versão original disponível em http://www.scielo.br/pdf/es/v30n109/v30n109a05.pdf

2 Para um detalhado informe sobre o surgimento do PISA e sobre as micropolíticas desse processo veja-se Morgan (2007)

3 Por exemplo, contrastando a natureza do trabalho e seus papéis típicos: “actividade física” versus “actividade mental”; “transformação de objectos materiais” versus “recolha de informação e resolução de problemas”; “tarefas prescritas e actividades rotineiras” versus “tarefas não rotineiras; “baixos níveis de interacção” versus “altos níveis de interacção”; “papéis resistentes à mudança” versus “papéis frequentemente e substancialmente redefinidos” (Weers & Kerchner, 1996, p. 146).

4 De acordo com Gagnon (1990, pp. 2-6), o modelo «racionalista» compreende os seguintes elementos centrais: os «dados e os resultados» das pesquisas constituem «um novo conhecimento», baseado em «cânones metodológicos especializados», produzido para dar fundamentação racional, objectiva, à decisão política e para resolver «problemas políticos» relevantes; por fim, compete aos experts serem capazes de mediar entre o conhecimento objectivo criado e os interesses políticos.

5 Trecho de entrevista a membro do Secretariado do PISA conduzida no âmbito da pesquisa sobre a fabricação supranacional do PISA (ver Carvalho, 2009)

Intensificação e sofisticação dos processos da regulação transnacional em educação 1

Luís Miguel Carvalho

Introdução

Os governos nacionais confrontam-se com o alargamento do campo de ação das organizações internacionais (OIs), cujos questionamentos e preconizações percebem ora como obrigações, ora como recursos indispensáveis para dirigir e administrar os sistemas educativos (Barroso, 2005). No caso da educação, essas organizações vêm alargando o seu âmbito de ação por meio da difusão de ideias e de normas relativas às estruturas, aos processos e às práticas educativas; mas vêm-no fazendo, ademais, por intermédio da criação de novos espaços e novos instrumentos para o governo da educação (Leuze; Martens; Rusconi, 2007; Rizvi; Lingard, 2006). Este artigo apresenta uma análise sobre os modos como as OIs vêm fabricando agendas para o governo da educação em nome de um conhecimento pericial. Concretamente, o artigo versa a intervenção da Organização para a Cooperação e o Desenvolvimento Económico (OCDE) por meio de um dos seus principais, senão o principal, dispositivos de avaliação dos sistemas escolares: o Programa Internacional de Avaliação de Estudantes (PISA).2

Nas duas últimas décadas, a intervenção da OCDE se vem materializando, mais e mais, por meio de abordagens que vêm tomando como principal forma a “comparação dos Estados, uns com outros, e face a critérios estandardizados” (Martens, 2007, p. 40), em vez de focarem, como antes, a situação de cada país per si, respeitando as especificidades nacionais. Essa “viragem comparatista”, seguindo os termos de Martens (2007), foi acompanhada pela criação e disseminação de instrumentos que a OCDE apresenta como visando apoiar a monitorização da qualidade e da eficácia dos sistemas educativos.

Entre outros dispositivos dessa natureza, sobressai o PISA, preparado no final da década de 1990 e ativado desde o início do presente século (Morgan, 2007), com regularidade trianual e sem interrupções. Apesar da contestação de que tem siso alvo, por razões de ordem epistemológica, metodológica, ou mesmo política, o PISA obteve uma consagração expressiva no campo educacional, onde vem servindo para legitimar medidas políticas, instigar pesquisas e animar debates públicos. É possível dizer que se trata de um objeto ao qual políticos, profissionais da comunicação social, investigadores, peritos e altos quadros da administração recorrem com frequência para escrutinar, problematizar ou apontar soluções para o setor da educação. Exatamente por isso, o PISA é aqui analisado como um instrumento participante nos processos de regulação, política e social dos sistemas educativos (Carvalho, 2009). Desse modo, o artigo procura mostrar que a presença do PISA em vários mundos sociais ligados à educação é uma manifestação da relevância atual da “regulação” (Dupriez & Maroy, 2003; Barroso, 2005) induzida por agências internacionais e exercida com base no conhecimento pericial. Em suma, o artigo aborda o PISA como um analisador da intervenção da OCDE na “regulação transnacional” (Barroso, 2005; Djelic & Sahlin-Andersson, 2006), por meio dessa avaliação internacional comparada.

A primeira parte do texto foca em dois fatores centrais das dinâmicas de regulação transnacional: como opera e que efeitos induz o PISA nos contextos nacionais. Depois, na segunda parte, o texto mostra como se vem intensificando e sofisticando a intervenção da OCDE na regulação transnacional da educação, dando para isso atenção a um conjunto de novas dinâmicas de alargamento da influência do PISA, por via da criação de novos modos de usar os dados por parte dos seus potenciais utilizadores, mormente pelos decisores políticos nacionais.

Processos e efeitos da regulação transnacional baseada no conhecimento

Relativamente processos da regulação transnacional, a análise a seguir desenvolvida parte de uma idéia central: a sua influência manifesta-se tanto no plano das idéias, enquadrando problematizações e preconizações sobre os sistemas educativos, quanto no plano da ação organizada, pela configuração das interdependências entre os atores que intervêm nos processos de regulação. Relativamente aos efeitos, a análise concentra-se na clarificação do que pode ser o “efeito de agregação” (Lascomes; Simard, 2011) que o PISA, enquanto instrumento de regulação, gera nos diversos contextos nacionais e culturais que atravessa.

Processos

No decurso dos processos de construção e difusão de conhecimento que ocorrem no âmbito do PISA, a OCDE tem de corresponder positivamente à perceção segundo a qual gera um saber especializado, livre de interesses particulares e de circunstâncias particulares e fundado em um relativo consenso científico (Noaksson; Jacobsson, 2003). A OCDE concretiza essa performance de organização pericial em três planos articulados: na criação de objeto singular e consolidação de um monopólio de conhecimento; na mobilização e coordenação de mundos sociais diversos; na produção e difusão de conhecimento utilizável por mundos sociais diversos.

Relativamente ao primeiro, convém ter presente que o PISA é herdeiro de uma tradição de conhecimento comparado que, desde os anos 1950, vem procurando identificar os fatores explicativos das performances dos sistemas educativos, para encontrar os bons sistemas e as soluções associadas à obtenção dos resultados mais satisfatórios (Carvalho, 2009; Lindblad; Pettersson; Popkewitz, 2015; Mons, 2007). Porém, o PISA igualmente redirecionou essa tradição, pois desenvolveu um conjunto de características que o diferenciam de antepassados e de concorrentes. Para além de algumas caraterísticas técnicas, importa fixar duas outras novidades (Morgan, 2007): (1) a garantia dada pela OCDE aos países participantes de integrarem o órgão de direção do programa, envolvendo-os portanto na determinação daquilo que pode ser estudado e escrutinado; (2) a criação de uma área de expertise específica em torno de um objeto singular e novo — a competência de literacia dos estudantes —, afastando-se das soluções postas em marcha anteriormente, prisioneiras dos currículos nacionais.

Esse trabalho de fabricação de um objeto singular resulta do labor intelectual que suporta a conceptualização e a diferenciação desse objeto de avaliação, mas resulta igualmente de um processo de “interessamento” (Callon, 1986) de pessoas e organizações. Quem são esses atores? O PISA integra formalmente investigadores e peritos de centros de pesquisa públicos e privados, profissionais da OCDE, decisores políticos e membros da administração da educação, que participam em atividades de inquirição, na elaboração de publicações, em reuniões e em atividades formativas. No que concerne, apenas, aos peritos envolvidos, o PISA depende de peritos em: matérias avaliadas (matemática, leitura, ciências) e suas didáticas, psicometria e psicologia social, além dos peritos associados aos estudos sobre a eficácia das escolas, à econometria e à análise de políticas públicas.

Nesse contexto, o conhecimento gerado no âmbito do PISA é um conhecimento com um carácter “transgressivo” e se “autoriza a si mesmo”.3 É transgressivo, relativamente à investigação convencional, porque o seu ponto de partida e o seu ponto de chegada são parcialmente definidos pelos representantes nacionais, não pelos investigadores ou pelos peritos. Autoriza-se a si próprio, pois a construção dos inquéritos, das análises e das publicações é feita por equipas e por organizações que coletivamente validam a sua própria perícia e que não fazem prova da sua qualidade no contexto das revistas especializadas nem da avaliação pelos pares.

No processo de produção de conhecimento transgressivo e que se autoriza a si mesmo, encontramos uma manifestação da regulação transnacional que se exerce sobre as relações entre conhecimento e política, e entre produtores de conhecimento e políticos: por um lado, as autoridades públicas outorgam o estatuto de experts a certos atores sociais, bem como selecionam e determinam que conhecimento é válido para ser usado no campo político; por outro lado, as autoridades aceitam que esses putativos especialistas sejam parceiros-chave no processo de construção de representações e de regras para coordenação, orientação e controlo do sector educativo; finalmente, as relações entre as autoridades públicas e os peritos nacionais e internacionais são geradas e geridas sob a égide da OCDE, a qual, depois de reunir essa variedade de atores em torno do estudo de um objeto singular, cria condições para que esses interajam regularmente, fixa e arbitra as relações entre esses diferentes mundos, de acordo com valores e regras por si determinadas.

Da mobilização do vasto conjunto de saberes e interesses referidos, gera-se um conhecimento que tem como principais ingredientes as performances dos estudantes e os perfis de competência de literacia de estudantes e países. Ocorre que o conhecimento criado no âmbito do PISA não se circunscreve a tais matérias, pois igualmente oferece análises sobre a variabilidade desses desempenhos em função de relações estabelecidas com fatores de contexto, como estatuto socioeconómico dos estudantes ou a composição social das escolas, e com fatores dependentes da decisão política nacional, como o grau de autonomia das escolas ou a estrutura dos currículos. Mais ainda: identifica fatores que devem ser tomados em consideração pelos atores políticos, como questões políticas cruciais, criando um guião facilitador da análise, da avaliação e da reflexão por parte dos políticos. Ora, essas recomendações versam tópicas que têm sido centrais na agenda da OCDE nas últimas décadas: autonomia das escolas, accountability, privatização, livre escolha da escola, melhoria da escola (Maroy, 2006). Ou seja, o inquérito transforma-se em um lugar de fixação de uma agenda política e de recorrente verificação do seu cumprimento. Assim, a partir dos dados postos à disposição por intermédio do inquérito, emerge uma extensa e relevante “zona de conhecimento cinzento” (Lindblad, Pettersson & Popkewitz, 2015), em que se misturam as interpretações dos resultados e a elaboração de recomendações e preconizações para os sistemas educativos nacionais; e que estabelece uma ponte com os decisores políticos e outros profissionais no campo da educação.

Mas para que eles efetivamente usem esse conhecimento, é necessário que o percebam como credível, do ponto de vista científico, mas também como sendo relevante e, sobretudo, manuseável (Lindquist, 1990). Por essa razão,no âmbito do PISA se observa a produção de uma variedade de publicações —relatórios finais de ciclo, relatórios temáticos, bases de dados, relatórios técnicos, vídeos, manuais metodológicos, etc. — preparadas para serem manuseados por diferentes públicos — políticos, administradores, investigadores, famílias, meios de comunicação social — enfim, passíveis de serem reaproveitadas e desenvolvidas por diferentes utilizadores (Carvalho; Costa, 2011). Essa “multiplicação de usos” é parte indissociável do exercício de uma regulação que pretende operar sobre a reflexividade dos atores.

E, suma, nos modos de intervenção da OCDE no âmbito do PISA encontram-se os traços fortes de uma regulação transnacional que opera, fundamentalmente, por meio da criação de idéias, da capacidade para conectar mundos sociais diversos na produção, na difusão e no consumo desse conhecimento, garantindo, paralelamente, a imprescindibilidade da agência que o inventou e tem a responsabilidade de geri-lo: a OCDE. Consequentemente, para se conhecer o poder regulatório da OCDE por intermédio do PISA, há de se ter presente os modos como este é acolhido, interpretado e usado nos contextos aos quais é dirigido.

Efeitos

A expressão “efeito de agregação” (Lascoumes; Simard, 2011) é útil para mostrar que o poder regulador do PISA se concretiza por meio da criação das dependências funcionais e simbólicas que cria; e que tal poder é observável na variedade de usos que desencadeia em contextos habitados por diferentes interesses e por diferentes convicções. É possível observar esse efeito a partir de uma literatura já consolidada sobre a receção e os efeitos do PISA em contextos europeus (Carvalho; Costa, 2011, 2016), da qual emergem três ideias fundamentais:1. são múltiplos os propósitos que acompanham o uso do PISA; 2. é possível falar de uma presença seletiva do PISA, em função das especificidades dos contextos que o acolhem; 3. o PISA tem uma presença recorrente e relevante no escrutínio dos sistemas escolares.

A multiplicidade de propósitos que acompanham a receção e o uso do PISA — dos seus resultados, analises e objetos — nos contextos nacionais pode ser sistematizada em três palavras-chave: legitimação, informação, idealização (Carvalho; Costa, 2016). No primeiro caso — legitimação —, observa-se o uso do PISA na validação de reformas educativas (instrumentos de avaliação nacionais) e na promulgação de muitas políticas concretas. Nesse âmbito, é particularmente significativa a variedade de políticas reportadas pela investigação, por toda a Europa, como tendo alguma ligação com a receção do PISA. A área curricular é, como esperado, a mais referida e está associada às medidas de alteração de programas e modos de organização dos cursos, ao reforço de currículos centrados em competências e às alterações nos horários escolares. Mas são referidas mudanças em muitas outras áreas: na avaliação dos alunos; em programas de ação prioritária visando os problemas das desigualdades e da segregação; na formação inicial e continuada de professores; na gestão e avaliação das escolas; e mesmo na restruturação da rede escolar. No segundo caso, o PISA é ainda tomado como fonte primária ou suplementar para a pilotagem dos sistemas educativos, ora compensando a falta de dados e informações produzidos nacionalmente, ora acrescentando e/ou confirmando tais conhecimentos, e, finalmente, surge associado à construção de ideários, projeções ou narrativas sobre a educação e a reforma da educação. A essa multiplicidade de usos associam-se processos de seleção de resultados e análises do PISA que decorrem de especificidades políticas e culturais dos contextos que os acolhem. É já bem conhecido que a presença dos resultados do PISA nos contextos nacionais pode manifestar-se de modos diferentes e com diferentes intensidades: variando entre a glorificação, a escandalização ou a indiferença aos resultados nacionais; ou entre a surpresa, o choque ou a autopromoção (ver, respetivamente, Steiner--Khamsi, 2003; Grek, 2009). Por outro lado, há evidentes processos de seleção relativamente às grandes áreas sobre as quais a OCDE faz as suas recomendações, sendo algumas consideradas e outras completamente ignoradas. 4 Finalmente, em certos países, o conhecimento efetivamente mobilizado nos debates públicos é muitas vezes reduzido a um número limitado de afirmações simplificadas, enquanto noutros países se verifica a produção intensiva de novos conhecimentos dirigidos aos atores políticos nacionais, combinando ingredientes científicos, técnicos e políticos de modo a explicar ou a ajudar a compreender no âmbito nacional os resultados do PISA, oferecendo, por vezes, traduções diferentes e competitivas dos resultados do PISA (ver, e.g., Grek; Lawn; Ozga, 2009; Mangez; Cattonar, 2009; Mons; Pons, 2009; Neumann et al., 2012; Pons, 2012).

Esse conjunto de observações sobre a presença do PISA nos contextos nacionais, pela mão de diferentes utilizadores (políticos e outros atores envolvidos nos debates públicos sobre educação, peritos e investigadores nacionais), permite dizer que os materiais providenciados pelo PISA são reinterpretados, tornando-se aceitáveis e eficazes para cada contexto social e cognitivo que os recebe. Tal variedade não é um sinal de fraqueza, mas é o principal sinal da força do PISA como instrumento de regulação, na justa medida em que as divergências verificadas no uso dado ao PISA coexistem com uma convergência para o uso do PISA. É esta manifestação do seu “efeito de agregação”: as controvérsias, as ficções ou os escrutínios dos sistemas, das políticas e das práticas educacionais são feitas em comprometimento com os seus resultados ou com os seus referenciais. Dito de outro modo, a força do PISA reside na sua compreensão e aceitação como um recurso que é percebido como natural e inevitável para a ação política no sector educativo. Em suma, não se trata de uma intervenção que exerce impacto direto sobre as políticas. De facto, o PISA, como outros meios da regulação transnacional, estabelece o quadro mais geral das possibilidades e das expetativas para as políticas nacionais e estabelece limites para elas, criando uma “meta-política”, como referem Dale e Robertson (2014).

Intensificação e sofisticação da regulação transnacional

De acordo com Sellar e Lingard (2014), a OCDE tem procurado aumentar a influência do PISA ao longo do seu ciclo de vida, uma “expansão” envolveu três tipos de alterações: a ampliação do espaço geopolítico do inquérito; o alargamento dos objetos e dos sujeitos que são avaliados pelo PISA; o alargamento do seu poder explicativo, isto é, a criação de novos modos de usar os dados de modo a alargar a influência do conhecimento proporcionado pelo PISA na decisão política. O texto centra-se, seguidamente, nessa última, de modo a enfatizar a intensificação e a sofisticação dos processos de regulação transnacional. E observa dimensões da manifestação dessa intensificação e sofisticação: a difusão de conhecimento, atendendo à ampliação da variedade de produtos gerados em audiências muito diversas; a troca de conhecimento, atendendo ao alargamento dos atores que a OCDE põe em interação, a partir do uso dos dados do PISA; a produção de conhecimento, atendendo ao aumento dos inquéritos sobre a literacia que vem desenvolvendo. Por razões de economia do texto, dar-se-á destaque a essa última dimensão.

Sofisticação dos produtos

Se em 2008, no contexto de uma pesquisa então em curso, foi possível detetar uma significativa variedade de publicações sob a marca PISA — relatórios de vários tipos, bases de dados, documentos metodológicos, cada um deles dirigido a populações-alvo específicas (Carvalho; Costa, 2011) —, na atualidade o cenário é diferente: há mais produtos com funções de estímulo à emulação e à aprendizagem política, mormente por meio de uma série de relatórios com estudos de certos países, sugestivamente sob o título “Strong Performers and Successful Reformers”. Há novos produtos que oferecem informação de rápido consumo, que em quatro ou cinco páginas dão respostas práticas sobre temas muito particulares e do interesse de administradores, formadores e professores, tendo por base resultados do PISA (sobre atitudes e desempenhos dos alunos, sobre problemas relacionados com os contextos familiares e até mesmo sobre escolhas políticas).

Há novos tipos de produtos que permitem aos utilizadores da plataforma informática criada pela OCDE a construção dos seus próprios modelos de comparações e relatórios tendo por base os dados do PISA. Essa tecnologia surge associada a uma plataforma sugestivamente designada de “Education GPS”. Trata-se de uma plataforma que suporta uma intervenção reguladora de novo tipo na OCDE, por meio da qual, como refere Williamson (2015, p. 12), a OCDE evoluiu de centro de cálculo para centro de visualização de “milões de observações e medidas” que ali estão “integradas e disponíveis para exame, análise, avaliação e comparação” pelos interessados.

Novos fóruns temáticos

Uma segunda modalidade da intensificação e da sofisticação dos processos de regulação transnacional, observáveis a partir do PISA, envolve a criação e estabilização de novos processos de validação de boas práticas e o estabelecimento de agenda por meio de fóruns temáticos. Uma manifestação particularmente interessante e a merecer mais estudo ocorre desde 2011, no quadro do International Summit on the Teaching Profession (ver Robertson, 2012).

Essa “cimeira” (o termo é, em si, significativo) resulta da cooperação, desencadeada

pela OCDE, com a Internacional da Educação (uma federação mundial de associações sindicais e de outras organizações representativas de trabalhadores da educação) e com as autoridades nacionais, especialmente das autoridades dos países onde se realiza. Cada cimeira reúne atores de sistemas educativos (políticos, altos funcionários, sindicalistas e peritos), os quais são selecionados por pertencerem a países que têm melhores performances no PISA ou por serem percebidos como países com melhorias rápidas. As atividades desenvolvidas nas cimeiras baseiam-se em relatórios elaborados propositadamente pela OCDE, mobilizando frequentemente os resultados do PISA e do Teaching and Learning International

Survey (TALIS), usados para selecionar e difundir exemplos de “políticas e práticas promissoras” que objetivam melhorar o ensino e o trabalho dos professores.5

Trata-se obviamente de um processo de construção de agenda, mas também de criação de um espaço organizado, municiado e arbitrado pela OCDE, para a consensualização entre atores fortes da regulação da educação.

Novas inquirições: uma relação direta com as escolas?

O alargamento das atividades de inquirição desenvolvidas no âmbito do PISA tem ocorrido pela sua extensão para outras populações-alvo, como os adultos e os estudantes do ensino superior — respetivamente, com o Programme for International Assessment of Adults Competencies (ver, e.g., Grek, 2010) e o Assessment of Higher Education Learning Outcomes — e para outros países, com o PISA for Development (ver Addey, 2016). Um outo dispositivo relevante e sobre o qual aqui recai a atenção deste texto é o PISA-based Test for Schools, por meio do qual a OCDE pode vir a ganhar contacto direto com os processos de governo e gestão das escolas, de algum modo flanqueando as autoridades nacionais. Sucintamente, esse novo instrumento integra um conjunto de testes semelhantes aos usados no inquérito PISA. Inclui, portanto, testes escritos e questionários e recorre às mesmas métricas usadas no PISA (de modo a gerar dados comparáveis).6 Ocorre que o teste é aplicado nas escolas de modo a produzir conhecimento sobre cada escola participante, sendo, ao final do processo avaliativo, devolvido; já o PISA é aplicado em escolas de um país de modo a avaliar o seu sistema educativo e a produzir conhecimento sobre esse sistema.

As investigações recentes sobre o PISA-based Test for Schools (Rutkowski, 2015; Lewis; Sellar; Lingard, 2016) permitem entender a adesão das escolas, em função de vários fatores: o acesso a dados sobre a sua performance e sobre os fatores de contexto e de orientação educativa; o acesso a propostas de boas práticas validadas por uma organização pericial; a possibilidade de introduzir e legitimar mudanças locais em função de evidências e das medidas quantificadas; a possibilidade de partilha de conhecimentos e recursos com outras escolas; a possibilidade de se apresentarem publicamente como instituições orientadas por “padrões” de qualidade mundial e, no caso de terem bons resultados, como fazendo parte de uma elite de escolas de classe mundial e um bem de consumo localmente apetecível.

Esse último fator indicia claramente a presença de uma lógica de certificação externa de qualidade como modalidade para a regulação da oferta e da procura escolar. É, aliás, neste plano — da reflexão sobre as implicações que uma eventual expansão desse teste possa ter para os processos de regulação da educação (e para a governação das escolas, em particular) — que importa refletir um pouco mais: a emergência e a eventual consolidação desse inquérito configuram uma intensificação dos processos de multirregulação da educação, com um reforço do protagonismo e da influência da regulação transnacional sobre a regulação feita pelas autoridades públicas nacionais.

Em primeiro lugar — e quase óbvio —, o PISA-based Test for Schools permite à OCDE um alargamento do seu espaço de observação e influência sobre as atividades educativas, estabelecendo relações diretas com as escolas e seus dirigentes e gestores, assim podendo fazer chegar as suas injunções, com recurso ao conhecimento, diretamente sobre os atores e os processos de governo da escola. É certo que os testes só podem ser realizados com autorização dos governos, mas a intervenção da autoridade nacional esgota-se nessa função.

Em segundo lugar, o programa ativa a presença, nos processos de regulação, de um conjunto significativo de atores não estatais: por um lado, as organizações de filantropia e outras organizações sem fins lucrativos que financiam o programa e as escolas; por outro lado, as organizações multinacionais e nacionais ligadas ao comércio educativo, nomeadamente ligado ao setor dos meios de comunicação social de massa ou das organizações públicas ou privadas da área da investigação e da formação, que são acreditadas pela OCDE para gerir o processo em cada país (o que significa administrar o teste, analisar os dados e produzir relatórios para cada escola, sempre sob supervisão da OCDE). Desse modo, reforça-se o peso da participação nos processos de regulação de atores situados fora da esfera governamental, da administração e, indiretamente, da própria OCDE.

Em terceiro lugar, a inscrição de cada escola em uma rede unida ao redor de dados e de representações educativas veiculadas pela OCDE poderá facilitar a presença, nas escolas, de orientações sobre o seu governo que não estão submetidas aos padrões nacionais e que podem ser argumentadas, legitimadas e até exigidas, em função da sua aproximação a um cânone universal e ao saber técnico-científico.

Nesse âmbito, pode constituir um importante recurso na disputa pela autonomia das escolas diante da autoridade pública e da regulação vertical. Não deixa, por isso, de ser uma relação construída a partir de uma nova heteronomia — ou seja, de uma nova dependência, já não em relação ao Estado, mas agora diretamente de organizações periciais — que se integra em um fenómeno mais amplo de reforço do papel dos especialistas e das chamadas políticas baseadas nas evidências.

Fecho

Em toda a configuração de relações descritas ao longo deste texto, a OCDE parece adquirir maior centralidade e capacidade de intermediação, por meio de um modus operandi institucionalizado: idealizando, agregando atores, supervisionando interdependências (ver Marcusson, 2004). A partir do PISA, parece claro que há, entre organizações (e entre atores), um espaço no qual são trocados recursos materiais, informacionais, financeiros, e mesmo recursos simbólicos, como o status e a reputação dos intervenientes. Trata-se de um espaço no qual, paralelamente, são estabelecidas relações de interdependências entre todas as agências e são colocados em circulação e legitimação modos específicos de entender o que é a educação e como deve ser a escola governada. E, assim, pede-se regresso ao lugar de início deste texto: a relevância atual de processos de uma regulação transnacional baseada no conhecimento, que opera no campo educacional e atinge relevância nos processos de governo da educação, em dois planos: no plano das ideias, enquadrando problematizações e preconizações sobre os sistemas educativos; mas igualmente no plano da organização das relações de interdependência entre os atores que intervêm nos processos de regulação da educação. O PISA, que ao longo de uma década e meia foi catapultado para a posição de actante significativo nesses processos, parece agora servir de porta de entrada para o alongamento de uma rede de interdependências no setor da educação, senão mesmo ligando-o a outros setores.

Referências

Addey, C. (2016). PISA for development and the sacrifice of policy-relevant data. Educação & Sociedade, 37(137), 685-706.

Barroso, J. (2005). O estado, a educação e a regulação das políticas públicas. Educação & Sociedade, 26(92), 725-751.

Bieber, T., & Martens, K. (2011). The OECD PISA study as a soft power in education? Lessons from Switzerland and the US. European Journal of Education, 46(1), 101-116.

Callon, M. (1986). Some elements of sociology of translation. In J. Law (Ed.), Power, action and belief (196-233). London: Routledge and Kegan Paul.

Carvalho, L. M. (2009). Governando a educação pelo espelho do perito. Educação e Sociedade, 30(109), 1007-1032.

Carvalho, L. M., & Costa, E. (2011). Fabricando o espelho do perito: A construção de conhecimento pericial no mundo PISA. In L. M. Carvalho (Coord.), O espelho do perito (pp. 41-76). V. N. Gaia: Fundação Manuel Leão.

Carvalho, L. M., & Costa, E. (2016). The praise of mutual surveillance in Europe. In R. Normand, & J.-L. Derouet (Eds.), A European politics of education (pp. 53-71). London: Routledge.

Dale, R., & Robertson, S. (2014). Global education policies. In N. Yeates (Ed.), Understanding global social policy (2nd ed., 209-235). Bristol: Policy Press.

Djelic, M. L., & Sahlin-Andersson, K. (2006). Transnational governance. Cambridge: University Press.

Dupriez, V., & Maroy, C. (2003). Regulation in school systems: A theoretical analysis of the structural framework of the school system in French-speaking Belgium. Journal of Education Policy, 18(4), 75-392.

Grek, S. (2009). Governing by numbers: The PISA “effect” in Europe. Journal of Education Policy, 24(1), 23-37.

Grek, S. (2010). International organizations and the shared construction of policy ‘problems’. European Educational Research Journal, 9(3), 396-406.

Grek, S., Lawn, M., & Ozga, J. (2009). PISA and the policy debate in Scotland. Sísifo, 10, 73-84.

Lascoumes, P., & Simard, L. (2011). L’action publique au prisme de ses instruments. Revue Française de Science Politique, 61(1), 5-22.

Leuze, K., Martens, K., & Rusconi, A. (2007). Introduction. In K. Martens, A. Rusconi, K. Leuze, K. (Eds.), New arenas of education governance (pp. 3-15). London: Palgrave.

Lewis, S., Sellar, S., & Lingard, B. (2016). PISA for schools: Topological rationality and new spaces of the OECD’s global educational governance. Comparative Education Review, 60(1), 27-57.

Lindblad, S., Pettersson, D., & Popkewitz, T. S. (2015). International comparisons of school results: A systematic review of research on large scale assessments in education. Stockholm: Swedish Research Council.

Lindquist, E. A. (1990). The third community, policy inquiry, and social scientists. In S. Brooks, & A. G. Gagnon (Eds.), Social scientists, policy, and the state (pp. 21-51). New York: Praeger.

Mangez, E., & Cattonar, B. (2009). The status of PISA in the relationship between civil society and the educational sector in French-speaking Belgium. Sísifo, 10, 15-26.

Marcusson, M. (2004). The Organization for Economic Cooperation and Development as ideational artist and arbitrator. In B. Reinalda, & B. Verbeek (Eds.), Decision making within international organisations (pp. 90-105). London: Routledge.

Maroy, C. (2006). Convergências e divergências dos modos de regulação numa perpectiva europeia. In J. Barroso (Coord.), A regulação das políticas públicas de educação: Espaços, dinâmicas e actores (pp. 229-244). Lisboa: Educa.

Martens, K. (2007). How to become an influential actor: The ‘comparative turn’ in OECD education policy. In K. Martens, A. Rusconi, & K. Leuze (Eds.), New arenas of education governance (pp. 40-56). London: Palgrave.

Morgan, C. (2007). The OECD Programme for International Student Assessment: Unraveling a knowledge network. Tese (Doutorado). School of Public Policy and Administration, Carleton University, Ottawa.

Mons, N. (2007). L’évaluation des politiques éducatives. Apports, limites et nécessaire renouvellement des enquêtes internationales sur les acquis des élèves. Revue internationale de politique comparée, 14(3), 409-423.

Mons, N., & Pons, X. (2016). The reception of PISA in France: A cognitive approach of institutional debate (2001-2008). Sísifo, 10, 27-40.

Neumann, E., Kiss, A., Fejes, I., Bajomi, I., Berényi, E., Biró, Z., & Vida, J. (2012). The hard work of interpretation: The national politics of PISA reception in Hungary and Romania. European Educational Research Journal, 11(2), 227-242.

Noaksson, N., & Jacobsson, K. (2003). The production of ideas and expert knowledge in OECD. Stockholm: Stockholm Centre for Organizational Research.

Nowotny, H., Scott, P., & Gibbons, R. (2001). Re-thinking science: Knowledge and the public in the age of uncertainty. Cambridge: Polity Press.

Pons, X. (2012). Going beyond the ‘PISA shock’ discourse: An analysis of the cognitive reception of PISA in six European countries, 2001-2008. European Educational Research Journal, 11(2), 206-226.

Rizvi, F., & Lingard, B. (2006). Globalisation and the changing nature of the OECD’s educational work. In H. Lauder, P. Brown, J. A. Dillabough, & A. Halsey (Eds.), Education, globalisation and social change (pp.129-145). Oxford: Oxford University Press.

Robertson, S. L. (2012). Placing’ teachers in global governance agendas. Comparative Education Review, 56(3), 584-607.

Rutkowski, D. (2015). The OECD and the local: PISA-based test for schools in the USA. Discourse: Studies in the Cultural Politics of Education, 36(5), 683-699.

Sellar, B., & Lingard, B. (2014). The OECD and the expansion of PISA. British Educational Research Journal, 40(6), 917-936.

Steiner-Khamsi, G. (2003). The politics of league tables. Journal of Social Science Education, 1, 1-6.

Williamson, B. (2015). Digital education governance: Data visualization, predictive analytics, and ‘real-time’ policy instruments. Journal of Education Policy, 31(2), 123-141.

1 Texto original: Carvalho, L.M. (2016). Intensificação e sofisticação dos processos da regulação transnacional em educação: o caso do PISA. Educação & Sociedade, 37 (136), 669-683. Versão revista. Versão original disponível em: http://www.scielo.br/pdf/es/v37n136/1678-4626-es-37-136-00669.pdf

2 De acordo com Grek (2009), no final da década passada o PISA representava cerca de 30% do orçamento da sua Diretoria de Educação.

3 Recorre-se a dois dos elementos centrais de uma transfiguração na produção de conhecimento

sobre o social, já retratada em literatura dedicada à análise das narrativas dos peritos contemporâneos (Nowotny; Scott; Gibbons, 2001).

4 Como mostram Bieber e Martens (2011), o PISA gera orientações e recomendações sobre matérias muito variadas: equidade; padrões e estruturas escolares nacionais; formação de professores; garantia da qualidade; autonomia da escola; investigação e estatísticas nacionais.

5 Ver http://istp2015.org/

6 Teve já aplicações piloto nos Estados Unidos da América, no Reino Unido, no Canadá e na Espanha. A OCDE espera alargar os países envolvidos. Para informações detalhadas, por parte da OCDE, consultar: https://www.oecd.org/pisa/aboutpisa/pisa-based-test-for-schools.htm

Seeing education with one’s own’ eyes and through PISA lenses 1

Luís Miguel Carvalho e Estela Costa

Introduction

Since the beginning of the century, the Program for International Student Assessment (PISA), an international comparative assessment of 15-year-old students’ competencies, best known by its acronym – PISA – has become a major tool for the intervention of the Organization for Economic Cooperation and Development (OECD) in the transnational and national policy processes of the education sector (see, e.g., Henry, Lingard, Rizvi, & Taylor, 2001; Martens, 2007; Rizvi & Lingard, 2006; Rubenson, 2008).

This paper focuses on the active reception of this policy tool in six European spaces, along its first three cycles, 2000, 2003, and 2006. It brings to the fore two seemingly contrasting trends: the divergent uses of PISA in specific contexts, and yet the attractiveness achieved by this ‘mutual surveillance’ tool in different social worlds. The analysis gives particular attention to what is called the ‘update of reference societies’ in the context of national receptions of PISA, and is seen as part of a composite process that involves domestic reasons, either related to current agendas for education, or to deep historical factors, and injunctions related to the PISA rationale and PISA objects (main reports, extensive reports, technical reports, databases, and assessment tasks).

The analysis developed draws on the results of six studies on PISA circulation and use – in Francophone Belgium (Cattonar, Mangez, Delvaux, Mangez, & Maroy, 2009), France (Mons & Pons, 2009), Hungary (Bajomi, Berényi, Neumann, & Vida, 2009), Portugal (Afonso & Costa, 2009), Romania (Rostás, Kósa, Bodó, Kiss, & Fejes, 2009), and Scotland (Grek, Lawn, & Ozga, 2009) – and also on an integrative reading of these (Carvalho, Costa, & Afonso, 2009). The studies were conducted under the scope of a large-scale research project on the role of knowledge in the governing of education and health systems, and their sources include a wide range of documents (OECD and national official documents, parliamentary debates, and media), as well as interviews with relevant actors (OECD officials, national policy-makers, researchers, and administrators).2

The paper starts with a brief presentation of the analytical perspective adopted here, namely regarding the inscription of PISA in the dynamics of the transnational circulation of knowledge and policies. In the second section, the paper focuses on some conspicuous differences regarding the way the Program is mobilized in the six European spaces mentioned above, and how those differences are related to cultural and political features of the reception contexts. In the third section, the paper describes the ‘reference societies’ – that is, societies taken as references for the reform of educational systems (see Schriewer, 2000; Sellar & Lingard, 2013) – that have been taken into account in national education debates about PISA in those six European spaces. Finally, in the last section, the paper stresses the cognitive and normative injunctions introduced by PISA in the reconfiguration of national/regional imaginaries, as well as in judgments about the positioning of those nations/regions in a world space.

PISA and the multidirectional flow of knowledge and policy

PISA is one of the various means of OECD’s intervention in education. Yet, one has to recognize its importance and singularity: with PISA the OECD is no longer dependent on statistical data created by national systems since it ‘generates its own data’ (Henry et al., 2001). Even more, thanks to PISA, the OECD does not only create data of its own, it also claims itself to be the provider of an expert-based independent framework for the monitoring and steering of school systems (Carvalho, 2012). Thus, throughout this paper, we tackle PISA as a ‘public policy instrument’ (Lascoumes & Le Galès, 2007) that combines comparative assessment techniques with a set of representations of education and a singular perspective about the (transnational and national) construction of educational policies and practices, in order to lead and shape social action in a given policy domain (see Carvalho, 2012; Pons & van Zanten, 2007). This way, we also assume the existence of an umbilical link between international comparative instruments for the assessment of students’ performances and contemporary policy-making. Indeed, mutual surveillance tools like PISA are resources for social actors, as they drive the way of constructing problems and policies (see Nóvoa & Yariv-Mashal, 2003). Their weight in national policies cannot be ignored, as recently recalled by Gita Steiner-Khamsi (2012): [I]n present times, policy-makers tend more to refer to international standards, rather than to a specific educational system when they advocate policy transfers. (p. 9)

We may thus understand the creation and exchange of these types of ‘knowledge for policy’ tools related to the ‘construction of multilateral spaces’ (Rutkowski, 2007), by international organizations, as a form they adopt to broaden their ‘scope of action’, by influencing national debates, aims, goals, and structures for/of education systems (Leuze, Martens, & Rusconi, 2007, pp. 9–10). Similarly, we work with the idea that public policies are processes made up of a complex set of actions and interactions carried out by multiple actors, in multiple spaces (Rizvi & Lingard, 2010). Therefore, to understand ‘traveling’ policy instruments (Ozga & Jones, 2006) developed in international organizations, we must identify the mediation processes of reinterpretation, negotiation, and contextualization that accompany their reception. Following Tom Popkewitz (2003), we also recognize that many categories circulating on a worldwide scale – and that carry the weight of authority of universal categories – have a local or regional origin.

Therefore, we look at the circulation of PISA as a part of multidirectional processes that involve reinterpretation, de-contextualization, and re-contextualization, and where national, local, regional, and international agencies intertwine. Thus, we adhere to Schriewer’s (2000, p. 327) advice regarding the need to analytically address a ‘weaving of opposites’, like ‘internationalization and indigenization’ or ‘supranational integration and intra-national diversification’. That is why, in this text, we keep with an interpretative tension between a vision which marks the emergence of ways of thinking and acting, in the field of education, that tend to unify our repertoire of opportunities and that may be associated with the emergence of a ‘new educational planetspeak’ (Nóvoa, 2002), and a polycentric vision of power, which enhances translations that accompany processes of policies and knowledge relocation, by cultural and historical circumstances (Popkewitz, 2003; Nóvoa, Carvalho, & Yanez, 2013).

Diverse mobilizations of PISA: contexts do matter

Studies on the impact of PISA in national policies acknowledge the variability of form, content, amplitude, and intensity of such effect (see, e.g., Bieber & Martens, 2011; Dobbins & Martens, 2012; Ertl, 2006; Greger, 2012; Grek, 2009; Gür, Çelik, & Özoğlu, 2012; Rautalin & Alasuutari, 2009; Rinne, Kallo, & Hokka, 2004; Sellar & Lingard, 2013; Steiner-Khamsi, 2003; Takayama, 2008).

The six studies under analysis also reveal very contrasting receptions and uses of this regulatory instrument in the six European contexts. For instance, while in France the PISA results were translated into alternative versions of what should be considered legitimate knowledge about the functioning of French education (Mons & Pons, 2009), in Hungary the PISA vocabulary and data became a kind of a master narrative for domestic educational policies (Bajomi et al., 2009). While in Francophone Belgium the Program operates, and is operated, in such ways that the regulating state becomes a deregulated state (Cattonar et al., 2009), in Portugal it functions as a substitutive national evaluation tool and a provider of information that legitimates governmental policies (Afonso & Costa, 2009).

Furthermore, the six studies also show that PISA objects and texts are mobilized to fulfill several purposes: to legitimize policy initiatives; to manage the policy agenda; to develop secondary research; and even to support the making or the improving of domestic regulatory instruments. Representations and rules that PISA bears are filtered, modified, and sometimes even contested. For instance, policies invoking PISA knowledge vary from country to country, and within each country (over time), regarding the proposed solutions and the goals to be achieved, as well as the targets for each measure (students, teachers, and families) or even the declared values (Carvalho, 2012).

Moreover, one might refer to the existence of a reduction/selectivity process because the tool is judiciously mobilized by national actors: data are more frequently used than the analysis and arguments provided by the Program; they tend to use it to legitimate agendas, strengthen priority purposes, reforms, and ‘isolated’ decisions, rather than to create ‘new policies’. Facing these elements, one may say that in the political sense there is a selective mobilization of certain ‘pieces’ of PISA knowledge, which enables recurrent public policy issues to go back to the public agenda or to gain more visibility. So, PISA knowledge is used by decision-makers to legitimate policy problems and agendas and not necessarily to find solutions.

The results from the six studies allow us to associate these differences with the specificities of the contexts that actively receive PISA. Not only do the political factors that are usually taken into account – like the agency of brokers and entrepreneurs or the agenda of ongoing reforms (PISA is seemingly more invoked when issues of accountability, school system monitoring, or changing conceptions of curriculum are at stake) – but also where policy and knowledge factors combine, such as the existence of national knowledge traditions and structures within the field of assessment; their philosophical, theoretical, or methodological proximity-distance regarding PISA international comparative assessment framework; or even the strength of national involvements in previous international studies and/or in OECD initiatives. The research examined here shows that in countries with tradition in the field of international assessment and with close ties with OECD projects – Belgium, France, and Hungary, for example – there was a more evident public and political debate about PISA, and that the tool was much more deeply scrutinized or mobilized by each national scientific community. There is variability both in the public reception – positive or negative – and in the degree of conceptual proximity (or of assessment tradition) of the national actors toward the assessment framework.

Thus, in parallel with the studies mentioned above that identified and described different responses to PISA results, and their numerous and varied effects on national policies, we highlight the diffusion and circulation of PISA knowledge to be associated with a possibility of undetermined outcomes, historically and contextually contingent. Thereby, when ‘evidences’ and analyses generated within PISA circulate through specific political and cultural contexts, they are subject to re-contextualization – either absorbed or adapted or silenced – but they can also help to create new meanings and rules for local action. As recently demonstrated by Sellar and Lingard (2013, p. 479), with their analysis of the reactions to Shanghai’s performance on the 2009 PISA in three countries – England, the USA, and Australia – the tool is used to ‘push internal reform agendas’ within these nations, as well as to foster a ‘post-ideological framing of education policy’, structured around the so-called (positive) evidence regarding what-works.

In short, it can be said that there is no automatic or inevitable convergence of policies as an effect of PISA. In sum, specific political and cultural factors impact the way PISA objects and texts are reinterpreted, and made acceptable and efficient for the specific socio-cognitive contexts where they circulate, that is, within different nations and schooling systems.

The ‘update’ of reference societies

The specificities of the reception contexts are also important for understanding the diversity of ‘reference societies’ invoked in domestic debates about educational systems in each PISA cycle. By trying to follow a particular form of ‘externalization’ (Schriewer, 2000) – the externalization by reference to the world situation – we observed the direction followed by the eyes of national actors about the ‘Others’, when they produce comparisons or when invoking these societies as examples (positive or negative). Such is the case, for instance, when we identify the presence of the ‘Other’ that is meaningful for reasons of political and cultural history, and the ‘Other’ with whom there are strong links of transference, dispute, and competition; good examples are the references to Germany in France, or the references to England in Scotland, or also to Germany in Hungary, or Flemish community in Francophone community (Belgium). These references to the ‘neighbor’ can be very different and bring about various informational and/or legitimacy supplements.

Yet, one might have to seriously consider the existence of more direct PISA effects regarding the societies taken as reference points for the reflection on education systems. This direct effect has multiple expressions. There is the presence of the ‘Other’ that ranks the top of the league tables – Finland embodied this type, repeatedly seen as a case for lesson drawing and becoming a pilgrimage spot for educational tourism. Shanghai, China took over this mantle in the 2009 cycle of PISA. There is also the ‘Other’ that is taken into account because they made progress, as with Poland, which has been looked upon as an example of a ‘good student’ for having improved results. And there is still a third type of ‘Other’, which includes countries that have been mentioned because of the effects that the so-called PISA shock produces in their educational self-reflections. This happened in France, where Germany, but also Japan and Austria, were considered role models of seriousness in the way they handled results; and it also occurred in Hungary where the Polish reform experience was interpreted as a proper way to act and respond, following the latter’s bad initial results in PISA.

But there is still an invocation of other societies taken as examples; we call them the ‘new buddies’. Some of these are examined because they are seen as sharing the same educational problems. For instance, in Francophone Belgium, where debates were dominated by the ‘problem of equity’, other European countries were also examined – France, Portugal, and Austria – because of their similar results regarding variation among schools and failure rates. Other ‘new buddies’ are chosen just because they have a radically different performance. For instance, in Hungary, the contrasting results achieved by the Swedish system regarding the variation of the results between schools was useful to legitimate the introduction of new policy measures intended to restrict school choice, and to foster the heterogeneity of the schools’ social composition. The Hungarian case also reveals the extent to which the choice of the ‘Other’ may happen in the scope of the perception of the sharing of meaningful social and educational difficulties. It is materialized by means of the creation of equivalences between the integration of immigrants, in Germany, and the social issue of the Romanian ethnic minority, in Hungary. Therefore, we can say that invoking the ‘Other’ is a discursive exercise that only means to show the best and, through them, to animate the belief, and hope in ‘progress’. However, it also serves to find ‘equivalents’ and ‘differences’ and thus to trace old/new identities and to strengthen the legitimacy of choices and measures to be taken.

The injunctions of a knowledge-policy instrument

Taking into account the previous descriptive elements, one might say the update of ‘reference societies’ is impacted by PISA injunctions. Our point is that PISA’s frames, questions, indicators, and analyses define where, what, and why policy players should look for to give supplementary sense and legitimacy to their actions.

First of all, PISA induces where to look. PISA persuades the selection of ‘reference societies’, through criteria intrinsic to international comparative assessments, pointing to the best performers and major improvers, or better policy learners. Furthermore, the Program facilitates the selection of multiple other reference points, because it fixes there evaluated each three years.

But when PISA locates reflection about education in such competitive space-time, it simultaneously develops specific politics of cognition. In fact, the competitive space is not only a space of countries. Rather, it includes countries or education systems, but it is a space where these are displayed accordingly with specific categories of perception and judgment. Such is the case, for instance, through the opposed categories enacted recurrently in the analysis developed in PISA, like performance versus costs, school knowledge world versus real knowledge world, performance/quality versus equity, or challenge versus supportive system cultures. Moreover, the relations between these categories are discursively connected to other issues recurrently present in the OECD policy agenda over the last two decades: school autonomy, accountability, privatization, involvement of multiple stakeholders in the management and financing of schools, school choice, and school improvement. So, it is not just a matter of where to look, but of what to see elsewhere. Finally, to see education with one’s own eyes and through PISA’s lenses means legitimating several OECD dicta about governing processes.

We emphasize three: the primacy of the rational and ‘based-upon-data’ model, opposed to ideological and/or opinion-based coordination; the ‘free acquiescence’ of decision-makers to be involved and support expert-based mutual surveillance as an effective practice; and the perception of systematic assessment of literacy – generated by an organization that performs as an independent expert knowledge provider – as a useful and trustworthy resource for the steering of educational systems. In the six examined studies, we have found these dicta naturalized in the credibility assigned to PISA and OECD by national policy-makers and other social actors who declare PISA to be useful to raise awareness, to help identify problems and provide good examples of reform, and to portray educational systems’ weaknesses and strengths (Carvalho et al., 2009). Yet, we may find the same scripts enacted not at a national level, but at a supranational level as well; a good illustration of this is the reaction to the PISA 2009 report by the European Commissioner for Education, Culture, Multilingualism, and Youth (European Commission, 2010). The reaction has it all: it has the praise of mutual surveillance (‘PISA identifies the most recent trends in education by assessing our performance compared with that of the most successful countries worldwide’); it has the identification of common problems in Europe (‘the gap between the best performing education systems, and between the best performing schools and pupils within countries, remains too big’); it has the tribute to the European ‘best performers’ (Finland as usual), but also the assurance that improvement is possible (‘member states like Poland, Greece, Portugal, Italy, Spain,

Romania, and Bulgaria where cited as having substantially improved their scores in certain areas’); and finally, it has the narrative of progress through competition (‘Europe must continue to improve if it is to maintain its status as a knowledge society’).

Hence, if PISA provides a means and opportunity to education systems for selecting ‘reference societies’, it also participates in the reconfiguration of the national and regional imaginaries and the judgment about their places in the space-world according to the international standards that PISA carries and fixes.

We thus argue that there is neither a direct transfer nor an automatic import, nor a convergence of policies as a result of the circulation and reception of PISA. Our point is that PISA fabrications open and close routes and establish a set of possibilities for public action for old and new policies. When mobilized in different sociocultural spaces, these fabrications are reworked by – and help to rewrite – systems of meaning and power relations, thus producing particular effects (Popkewitz, 2003). However, PISA becomes an unavoidable (and ‘obvious’) provider of information ‘based on proof’, a tool that creates and allows the creation of new problems and imagined new tomorrows.

References

Afonso, N., & Costa, E. (2009). Study on the use and circulation of PISA at the national level. The reception of PISA in Portugal. Project KNOWandPOL, WP12. Retrieved October 29, 2013, from http://www.knowandpol.eu/

Bajomi, I., Berényi, E., Neumann, E., & Vida, J. (2009). The reception of PISA in Hungary. Project KNOWandPOL, WP12. Retrieved October 29, 2013, from http://www.knowandpol.eu/

Bieber, T., & Martens, K. (2011). The OECD PISA study as a soft power in education? Lessons from Switzerland and the US. European Journal of Education, 46(1), 101–116. doi:10.1111/j.1465-3435.2010.01462.x

Carvalho, L. M. (2012). The fabrications and travels of a knowledge-policy instrument. European Educational Research Journal, 11, 172–188. doi:10.2304/eerj.2012.11.2.172

Carvalho, L. M., Costa, E., & Afonso, N. (2009). PISA—Fabrication, circulation and use in six European Countries: Integrative report. Project KNOWandPOL, WP17. Retrieved November 5, 2013, from http://www.knowandpol.eu/

Cattonar, B., Mangez, E., Delvaux, B., Mangez, C., & Maroy, C. (2009). Usages et circulation des enquêtes PISA en communauté française de Belgique [Uses and circulation of PISA in Belgium’s French community]. Project KNOWandPOL, WP12. Retrieved October 29, 2013, from http://www.knowandpol.eu/

Dobbins, M., & Martens, K. (2012). Towards an education approach à la finlandaise? French education policy after PISA. Journal of Education Policy, 27(1), 23–43. doi:10.1080/02680939.2011.622413

Ertl, H. (2006). Educational standards and the changing discourse on education: The reception and consequences of the PISA study in Germany. Oxford Review of Education, 32, 619–634. doi:10.1080/03054980600976320

European Commission (2010). Communicating research for evidence-based policymaking: A practical guide for researchers in socio-economic sciences and humanities. Luxembourg: Publications Office of the European Union.

Greger, D. (2012). When PISA does not matter? The case of the Czech Republic and Germany. Human Affairs, 22(1), 31–42. doi:10.2478/s13374-012-0004-5

Grek, S. (2009). Governing by numbers: The PISA ‘effect’ in Europe. Journal of Education Policy, 24(1), 23–37. doi:10.1080/02680930802412669

Grek, S., Lawn, M., & Ozga, J. (2009). Study on the use and circulation of PISA in Scotland Project KNOWandPOL, WP12. Retrieved October 5, 2013, from http://www.knowandpol.eu/

Gür, B. S., Çelik, Z., & Özoğlu, M. (2012). Policy options for Turkey: A critique of the interpretation and utilization of PISA results in Turkey. Journal of Education Policy, 27(1), 1–21. doi:10.1080/02680939.2011.595509

Henry, M., Lingard, B., Rizvi, F., & Taylor, S. (2001). The OECD, globalisation and education policy. Oxford: Pergamon Press.

Lascoumes, P., & Le Galès, P. (2007) Introduction: Understanding public policy through its instruments — From the nature of instruments to the sociology of public policy instrumentation. Governance, 20(1), 1–21. doi:10.1111/j.1468-0491.2007.00342.x

Leuze, K., Martens, K., & Rusconi, A. (2007). New arenas of education governance: The impact of international organisations and markets on education policy making. In K. Martens, A. Rusconi, & K. Leuze (Eds.), New arenas of education governance (pp. 3–15). London: Routledge.

Martens, K. (2007). How to become an influential actor: The comparative turn in OECD education policy. In K. Martens, A. Rusconi, & K. Leuze (Eds.), New arenas of education governance (pp. 40–56). New York, NY: Palgrave Macmillan.

Mons, N., & Pons, X. (2009). The reception of PISA in France. Knowledge and regulation of the educational system. Project KNOWandPOL, WP12. Retrieved October 14, 2013, from http://www.knowandpol.eu/

Nóvoa, A. (2002). Ways of thinking about education in Europe. In A. Nóvoa, & M. Lawn (Eds.), Fabricating Europe: The formation of an education space (pp. 131–155). Dordrecht: Kluwer Academic Publishers.

Nóvoa, A., Carvalho, L. M., & Yanez, C. (2013). La comparaison comme politique: Le tournant du début du XXIe siècle [Comparison as policy: The turn of the twenty-first century]. Manuscript submitted for publication.

Nóvoa, A., & Yariv-Mashal, T. (2003). Comparative research in education: A mode of governance or a historical journey. Comparative Education, 39, 423–438. doi:10.1080/0305006032000162002

Ozga, J., & Jones, R. (2006). Travelling and embedded policy: The case of knowledge transfer. Journal of Education Policy, 21(1), 1–17. doi:10.1080/02680930500391462

Pons, X. (2012). Going beyond the ‘PISA shock’ discourse: An analysis of the cognitive reception of PISA in six European countries, 2001–2008. European Educational Research Journal, 11, 206–226. doi:10.2304/eerj.2012.11.2.206

Pons, X., & van Zanten, A. (2007). Knowledge circulation, regulation and governance. Project KNOWandPOL, Literature Review. Retrieved October 7, 2013, from http://www.knowandpol.eu/

Rautalin, M., & Alasuutari, P. (2009). The uses of the national PISA results by Finnish officials in central government. Journal of Education Policy, 24, 539–556. doi:10.1080/02680930903131267

Rinne, R., Kallo, J., & Hokka, S. (2004). Too eager to comply? OECD education policies and the Finnish response. European Educational Research Journal, 3, 454–486. doi:10.2304/eerj.2004.3.

Rizvi, F., & Lingard, B. (2006). Globalisation and the changing nature of the OECD’s educational work. In H. Lauder, P. Brown, J.-A. Dillabough, & A. Halsey (Eds.), Education, globalization and social change (pp. 247–276). Oxford: Oxford University Press.

Rizvi, F., & Lingard, B. (2010). Globalizing education policy. London: Routledge.

Rostás, Z., Kósa, I., Bodó, J., Kiss, A., & Fejes, I. (2009). Use and circulation of PISA in a Romanian context. Project KNOWandPOL, WP12. Retrieved October 29, 2013, from http://www.knowandpol.eu/

Rubenson, K. (2008). OECD education policy and world hegemony. In R. Mahon, & S. McBride (Eds.), The OECD and transnational governance (pp. 242–259). Vancouver: UBC Press.

Rutkowski, D. J. (2007). Converging us softly: How intergovernmental organisations promote neoliberal educational policy. Critical Studies in Education, 48, 229–247. doi:10.1080/17508480701494259

Schriewer, J. (2000). World-system and interrelationship-networks. In T. S. Popkewitz (Ed.), Educational knowledge (pp. 305–342). Albany: State University of New York.

Sellar, S., & Lingard, B. (2013). Looking East: Shanghai, PISA 2009, and the reconstitution of reference societies in the global education policy field. Comparative Education, 49, 464–485. doi:10.1080/03050068.2013.770943

Steiner-Khamsi, G. (2003). The politics of league tables. Journal of Social Science Education, 1, 1–6.

Steiner-Khamsi, G. (2012). Understanding policy borrowing and lending: Building comparative policy studies. In G. Steiner-Khamsi, & F. Waldow (Eds.), World yearbook of education 2012: Policy borrowing and lending in education (pp. 1–17). London: Routledge.

Takayama, K. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44, 387–407. doi:10.1080/03050060802481413

1 Texto original: Carvalho, L. M., & Costa, E. (2015). Seeing education with one’s own’ eyes and through PISA lenses. Discourse: Studies in the Cultural Politics of Education, 36(5), 638–646.

2 For a different analysis of these six studies, focusing the knowledge actors mobilize when they

talk about PISA, in the public debate, see Pons (2012).

Fifteen years looking at the mirror: on the presence of PISA in education policy processes (Portugal, 2000-2016) 1

Luís Miguel Carvalho, Estela Costa e Catarina Gonçalves

Introduction

PISA was like a mirror. Each time the results of PISA came out, the country saw itself in the mirror. In comparison with other people who were in the picture, we were saying, ‘Well, we are not as tall as we thought, not as skinny or as fat or as handsome as we thought. We need to work to get taller and look nicer in this image.’ (Former Minister of Education, 2012).2

The use of the mirror metaphor do describe the reception of the PISA data at the national level takes the first few seconds of a video by OECD and Pearson Foundation on the Portuguese education system, as part of the series Strong Performers and Successful Reformers in Education.3 It is worth noting that this is a metaphor that has often been used by those who are in charge of PISA:

(…) we are more like a mirror. We have a mirror; you can look at it and you can see how the world looks (…) (OECD official, Interview, 2009, apud Carvalho & Costa, 2009, p.137)

PISA shows what is possible in education, and it helps countries see themselves in the mirror of the education opportunities and results delivered by the world’s leaders in education (OECD’s Deputy Director for Education and Skills, 2013).4

From an analytical point of view, the power of this ‘mirror’ which OECD cyclically brings back to national politicians’ sight resides in what it aims to display (students’ performances in tests as indicators of school systems’ quality-equity), in the credibility it holds and which it employs to attract those who might wish to look at themselves and to be seen through it, but also in the multiplicity of uses that it is actually given (Carvalho, 2014).

Indeed, the reference to these multiple and recurring uses is present across the literature on the reception and effects of PISA in the national European contexts (Carvalho & Costa, 2016): there is an assignation of multiple purposes to PISA texts and objects, particularly of legitimation (of reforms, policies, instruments), information (either as primary or as complementary source for the steering of education systems), and idealization (supporting the construction of diverse educational ideals or narratives); PISA objects/texts are re-interpreted, made acceptable and efficient for each socio-cognitive context, thus, subject to diverse selections (e.g. regarding the policy domains addressed by PISA recommendations); despite the persistent existence of critical voices, PISA is usually perceived as capable of raising public awareness, of identifying problems and providing good examples, of accurately portraying the weaknesses and strengths of national systems, and such perceptions come together with the adoption of evidence-based approaches, even if in some cases as a phony adoption or as part of a categorical script for policy-making.

The paper describes and discusses the proximity/distance of the Portuguese case in face of the identified tendencies and the continuities/discontinuities in the way actors who are involved in the public policies relate to PISA. The paper questions what happens when the knowledge for policy generated within PISA gets in contact with its political audience and/or when the latter calls upon the former. What have the policy actors been doing with PISA data and analysis when they consider, express and justify their own choices? How do they look at ‘the mirror’, how do they observe themselves in it and what do they say about what they see, each time OECD turns the polished face of PISA to the national spaces?

The paper draws on research conducted by one of its authors on the debate held by politicians in Portugal on the contexts of the first three PISA cycles (2000, 2003, and 2006), particularly on interviews conducted with three Ministers of Education (Afonso & Costa, 2011), and research conducted more recently by the authors on how policy-makers relationships with PISA have evolved in the surveys that followed (2009, 2012, and 2015), using as major sources formal and informal policy documents (laws, government decrees, ministerial speeches, official press releases, information displayed in the official websites) and, for the analysis of the reception of PISA 2015, a collection of texts published in the written press.

In the first section, the paper focuses on how PISA is received, considering the debate raged around it and how this has evolved since 2001 to the 2012 PISA edition. Two interpretative lines are developed: the consolidation of PISA’s credibility as a source for the production of statements on the educational system and policies; the emergence of new actors and modes of intervention in the production of knowledge for national policy drawing on PISA. In the second section, the paper presents an exploratory analysis on the reception of PISA 2015 in the Portuguese press, following the release of the programme’s last cycle results. Two interpretative lines are developed: the reception of the 2015 data as a moment for the consecration of PISA’s credibility; the practices of qualification and disqualification of educational policies and perspectives, which emerge as the main common elements of that reception. The paper concludes with the main points from the analysis developed in each section.

PISA reception in Portuguese policy processes: two trends

We analyse the trajectory of PISA’s political reception in Portugal according to two descriptive and interpretative lines: (a) from an unsure participation to a legitimating mobilisation; (b) the difficult transformation of comparative knowledge for policy into knowledge for national policy.

From an unsure participation to a legitimating mobilisation: signs of consolidation

The reception of PISA in the political processes in Portugal was marked by an initial unsure participation, more so for reasons related to political circumstances than as a result of a rational valorisation of this specific assessment instrument (Afonso & Costa, 2011). Indeed, the first participation in PISA happened in a context in which the consideration of student performance as an indicator of the system’s quality had just been incorporated in the educational agenda, and also in the aftermath of an undisputed negative perception of the Trends in International Mathematics and Science Study (TIMSS) results, published in 1996.

Doubts and debates around the decision to participate in PISA are perceptible in this recollection by the Minister of Education who was involved in the first PISA cycle:

Initially there was [in the Ministry] on the part of some sectors a reaction, I would say, negative regarding PISA. (…) I do not say that there were large sectors who were against an international comparison, what I think is that maybe there was the idea of: “First, let’s try to create conditions and solve some problems that are apparently easy to solve, and then we move forward” (Former Minister of Education, Interview #1, 2008).

More recently, in a so-called activity report by the XIX Portuguese Government [2011-2015], PISA results were used in a very different way. The document presents, right in the first pages, a figure with no description or analysis whatsoever (Fig.1) – therefore capable of speaking for itself –, in which the variation of PISA results is shown as the series in function of which the reader shall intuit/judge the effect of educational policies of governments with different orientations: political measures taken by centre-left governments (1995-2001, 2005-2011), namely targeting the curriculum with specific programmes for the improvement in maths and reading, from 2006 on; the introduction of national exams, namely in the ninth grade, which was accomplished by those governments but was in fact legislated by previous centre-right oriented governments [2001-2005] and which presence was reinforced by the departing government [2011-2015].

Figure 1: PISA results as a policy barometer. Source: MEC, 2015, p. 23

From the hesitation at the time of adhesion to the implicit acceptance of the performance of the Portuguese students, who were selected to be included in the assessed sample group, as the reference indicator for the legitimating/‘dislegitimating’ of policies, there is a consolidation of PISA in Portugal, especially from 2005 on. We identify two indicators that reasonably support this statement: the persistent use of PISA outcomes to justify the adoption of public policies; the activation of OECD scripts on the meanings and processes for the governing of education, in the context of enunciations in which policy-makers justify or preconize the use of PISA.

Regarding the first, it is possible to display a large list of policy areas in which PISA has been invoked (Table 1). Some of these interventions are not new in the Portuguese system: some, namely regarding Mathematics, Science and the national language, may be considered to be in line with political measures of the nineties (Fernandes, 2014). This does not blur, but on the contrary reinforces, the notion that the invocation of international expertise associated with PISA emerges as a relevant resource for the “certification” (Steiner-Khamsi, 2004) of interventions in the national policy processes.

Table 1: Policy proposals invoking PISA

Areas

Policy proposals

Year

Curriculum

The Mathematics Action Plan

The National Reading Plan

Educational territories of priority intervention program (re-launch)

The ‘Full-time school’ program

The Programme and Curricular targets of Mathematics

The Programme and Curricular targets of Portuguese

2005

2006

2006

2008

2013

2014

ICT

The Technological Plan

2007

Initial and in-service teacher training

The Programme for Training in Experimental Teaching of Science for Primary School Teachers

The National Programme for Portuguese Language Teaching in Primary Education

2005

2006

Regarding the second, it is possible to observe the enactment of the dicta about the (rational) decision maker who rules focusing on data about the outputs and outcomes of the system, with/through a comparative reason, and within a global cooperative and competitive space. The following quotes illustrate this:

an instrument which makes you think things differently, i.e. ‘be careful we are not alone’ and ‘results count’ (…) It is essential, mainly, to give us instruments of monitoring, and evaluation and, above all, to know how to put the problems, what we can get, what we cannot get, what is important, what is not important. (…) We had educational system performance variables – dropout rates, level of schooling, approval and retention rates, etc., – (…) but then we lacked the output in terms of knowing the quality of education. It means moving from the idea of quantity of education to the quality of education (…)”. (Former Minister of Education, Interview #4, 2009)

[PISA] allows you to have knowledge about the results, the acquired ones, the competences, the quality of what is taught, of what one learns, with the international cooperation. It allows countries to realize that they have similar problems, allows countries to have common strategies. The field of education has always benefited greatly from international comparisons, because it is what enables countries to carry on updating their ambition (Former Minister of Education, Interview #3, 2008).

Before PISA, we did not know exactly at what point we put in relation to the rest of the world. (Former Minister of Education, Visão [weekly magazine], 14/09/2014)

We cannot run the risk of stagnation. We need to move forward. We aim to be in the front group, where the Asian countries are but also many European countries such as Ireland, Austria, Germany, Belgium or Finland. (Former Secretary of State for Basic and Secondary Education, Público [Daily newspaper], 4/12/2013).

Together with the script on the national political action oriented by results (and in particular by the results of PISA), a narrative on improvement, aligned with the OECD’s visions, also emerges:

(…) because the goal of PISA (...) is to gradually bring us all closer to the higher levels, is to improve the averages so that people know more, go further in their education, have greater skills, have greater knowledge (…)”. (Former Minister of Education, Interview #1, 2008).

Our commitment to this process [joining PISA] is identical, not only in relation to PISA, but also by a process that opens then within the European Commission, when the European Commission wants to make a benchmarking in education (…) What interests me is that this benchmarking takes into account at least some of the variables: the way we progress and not how we are, or in other words, to introduce a dynamic and prospective analysis of the evaluation of education systems (…) the meaning of progression, that is, we were not interested in knowing if we are in 1st, 2nd or 3rd, if this is bad or worse, but how we are dealing with the problem, i.e., how we are overcoming the difficulties (Former Minister of Education, Interview #4, 2009).

(…) for a process of permanent quality control, the role played by international testing such as PISA is important. By allowing the construction of comparative databases, they also contribute to the international production of benchmarks that serve as parameters for schools and countries to improve the quality of their work” (Former Minister of Education, Governmental Website, 2007).5

In short, the enactment of the script of the education system internal improvement nurtures narratives that ascribe hope and belief to the steering of the systems on the basis of evidence.

The difficult transformation of comparative knowledge for policy into knowledge for national policy: signs of change

Besides the assignation of legitimation and idealization purposes to PISA texts and objects, policy-makers also present it as a primary or a supplementary source of information, as the following quotes exemplify:

Obviously, I never thought that PISA was a kind of magic wand that would solve the problems of education, but it was an element that could bring us a lot of information, even for us to know a little better what we were doing (…)” (Former Minister of Education, Interview #1, 2008).

The PISA study, this comparison, in a way, awakened those who had some responsibility and gave them an instrument, so to speak, to be able to support the decisions. (…) Now, I still think that, above all else, the effect of the publication of the results was important to demystify some ready-made thoughts. (… ) It is important when I say that this process of international studies and of PISA was, above all, a leverage of reflection, of an awareness. Now, before taking measures I was not going to study the PISA report first, of course. (...) I think politics is valued if it can be based on substantiated information. (Former Minister of Education, Interview #4, 2009).

Then there are those who do not decide without information and those who consider information irrelevant to decide. (…) What I think is that – and that is why I said it’s a very personal view and there may be some professional deformation here [sociologist] – I find it impossible to decide without having the necessary information to do so (…). I studied and analysed data and made a series of policy decisions based on PISA results, to which I added a few other elements which, however, I had asked to be collected and which seemed to me very important”. (Former Minister of Education, Interview #3, 2008).

The above quotations induce us to consider there is a gradation in the ways PISA is mobilised, from a resource to be consulted to a resource to be studied. And they also illustrate different types of political mobilisation of knowledge (Delvaux, 2009): to attest the need for a policy solution to a problem; to guide the preconisation and implementation of policies; to work as a sort of informational up-grade that keep actors updated on the changes in the sector in which they intervene.

This kind of statements also remind the hypothesis that informational mobilisations tend to vary according to actors’ dispositions towards evidence-based policy-making approaches, and depending on them having ad-hoc or formal structures that are able to work with this kind of data (Pons, 2012, Carvalho, 2014). However, on the basis of the data we present here, it is only possible to identify one common trait in the dispositions of the interviewed ministers: the recognition of the indispensability of this type of information for political decision.

In fact, the absence/need of appropriate information is a key-issue for some of the interviewees:

PISA helps to reflect and understand better, but it is not enough, we needed to complete our information and our knowledge from what PISA reveals, with finer, more in-depth studies (…) (Former Minister of Education, Interview #3, 2008).

(…) the education system remains unvalued and this subjects us to the international criteria and to the international studies due to the lack of self-evaluation. And one of the things I was worried about was, in particular, to make it possible for the National Education Council (CNE)6 to develop – although it had neither the money nor the know-how nor the competence to do so – but that it was an institution outside the Ministry that could ensure such an evaluation system, that is, it could even be the Ministry to provide the statistics, provide all the information, the indicators, etc., but there should be a reading outside the structure, by the various [social partners] (Former Minister of Education, Interview #4, 2009).

Regarding the production of information on the Portuguese education system based upon PISA results and databases, it is worth noting that there has been in Portugal a regular production of reports for all the editions. Moreover, it is possible to describe a transformation on the profile of PISA National Project Managers (NPM) in Portugal (Table 2): from qualified officials at the Ministry of Education (profiles in the first three cycles) to academics with high qualifications in areas other than Education and always with further research specialisation in the field of statistics (last two cycles).

Table 2: The national management of PISA

Cycle

Entity responsible for the National Management of PISA

NPM: Main area of activity along the professional path

NPM: Area of knowledge (highest qualification) / area of expertise

2000

Central State Administration: GAVE *

University and Central State Administration:

Psychology / Evaluation

2003

Central State Administration: GAVE

Business management

Management (IT)

2006

Central State Administration: GAVE

Business management

Management (IT)

2009

Central State Administration: GAVE

State Administration

Sociology**

2012

Central State Administration: ProjAVI//DGEE***

University

Psychology / Statistics

2015

Indirect State Administration: IAVE****

University

Physiology and Plant Biochemistry /Statistics

* GAVE (Educational Evaluation Office) was a structure at ME in which the PISA National Centre was settled until 2013. Three reports were written by GAVE (GAVE, 2001, 2004, 2007).

** In 2009, GAVE produced only one synthesis (GAVE, 2010). A more extensive report was prepared, outside the GAVE, by a team coordinated by a qualified sociologist in quantitative methods (Carvalho, H. et al., 2011), following a protocol between the ME and a research centre.

*** ProjAVI – Project Group for the International Evaluation of Students – was created within the scope of the ME with the mission to coordinate, in Portugal, the international projects in the field of educational evaluation, having produced a report on PISA (ProjAVI, 2013).

**** IAVE (Educational Evaluation Institute) is a structure integrated in the indirect administration of the State (endowed with pedagogical, scientific, administrative and financial autonomy) which arises following the extinction of GAVE. It generated reports on PISA 2015 (Marôco, 2016)

A parallel trend is perceptible regarding the publication of secondary analyses based on PISA data, in magazines and in the records of events that are expressly dedicated to the analysis of the Portuguese results in PISA, by authors who are associated with research centres and national universities, since 2009 (Table 3).

Table 3: Fields of expertise of the authors of secondary analyses based on PISA data (2009.-2016), considering the first author

Period

Publications in Journals

(nº of articles)

Publications in 2 Seminar Books

(nº of chapters) *

2009-2012

Sociology (1)

Education (2)

Economy and Management (4) **

Economy and Management (1) **

2013-2016

Economy and Management (3)

Education (1)

Economy and Management (5)

* Both publications are written by CNE, the event organiser (CNE, 2010 2015). In the first, only one of the chapters presents a secondary analysis. The following chapters, of a different nature, are written by authors related to the field of education.

** Member of a research centre of Mathematics Applied to Economics.

This data illustrates the emergence of actors who hold expertise on qualitative methods, matching that which is required by the models that orient the production of PISA data, and/or who are willing to produce policy knowledge, especially drawing on problems and approaches from the fields of Economics and Management (in some cases also Sociology and Psychology). This does not mean there has been an interruption, nor some sort of overshadowing, of the productions generated in the field of educational research. It instead represents the continuity of the lack of secondary analyses based on PISA data generated in the field of educational research in Portugal, detected up to 2009 (Costa, 2011), which in turn relates to the absence of a strong tradition of quantitative studies in the educational research in Portugal.

What may be of significance in all this, in terms of policy processes, are the possibilities opened up by the presence of more information directed towards policy spaces and, with that, of new frameworks (other worldviews on education and its governing), on the basis of which the original PISA data is translated for policy actors and is likely to be mobilized therein as a source of valuable informational elements.

Another (recent) phenomenon related to the presence of PISA in the policy processes is the emergence in Portugal and along 2015 and 2016, at the CNE7, of a project (and small-scale infrastructure) that seems to be imagined to become a mediator between PISA knowledge and the public and the decision-makers.

Named aQeduto8: Evaluation, quality and equity in Education, born by the initiative of the current leadership of the CNE and sponsored by the CNE and by a national philanthropic foundation9, the project is deliberately oriented towards the production of information from PISA data-bases about the Portuguese student’s results for policy-makers and lay persons. It intends not only to build “credible and sustained information” but also “explanations” regarding the “variation of the results” of Portuguese students in the PISA tests, “in simple language but preserving scientific rigour”.10 Its coordinator is a former NPM (2012-2013) who has expertise on multivariate data analysis. The project team includes four other members – the President of the CNE (researcher and professor of Sociology in a public University, and former Minister of Education [2001-2003]), a coordinator for data analysis who studied Economics, another researcher and professor of Sociology in a public university, and a member of the scientific and technical advisory team of the CNE.11 Thus, its composition is marked by actors that travel between different worlds (policy, administration, university) and relate to PISA developing work of varied natures (research and analysis, national management). The team is permeated by quantitative approaches to Psychology and Sociology, and by Economics, which in turn exemplifies the importance data and data sciences are achieving in the production of knowledge on education, in Portugal as elsewhere (Ozga, 2011).

Along the past year, the project has produced small pieces of secondary analysis on a set of ten themes on a monthly basis (e.g. Does grade repetition improve learning outcomes? What makes a good school?) coupled with several exchange and publication activities. These practices of continuous intermediation between PISA texts and the world of politics and public action, by a collective actor that was until recently simply playing its appointed role, producing recommendations and promoting consensus, may well be a sign of new phenomena in the national reception of PISA: the activation of the role of the knowledge-policy broker, as there is the intention of translating knowledge that is carried from one social world to another, and in that way and at the same time connecting those worlds (Delvaux, 2009); the activation of the role of the knowledge entrepreneur, as there is the intention of creating an organisation through the mobilisation of various resources and actors, around a specific agenda, in which experts play a central role (Van Zanten & Pons, 2006).12

The political reception of PISA 2015: an exploratory incursion through the press

In this section we develop an exploratory analysis on the reception of PISA 2015 in the Portuguese press. More precisely, this is an analysis of the interventions by political actors in the Portuguese written press published daily and weekly (not an analysis of the press coverage of the PISA 2015 results). Thus, we only consider texts that are published as an expression of its author’s opinion and we only cover a one-week period, starting on 6 December, the date of the public presentation of results (Table 4).

Table 4: Corpus

Newspaper

Title

Author

Correio da Manhã (7/12, p.31)

Great news

Mayor of Lisbon / Newspaper Columnist

Diário de Notícias (7/12, p.11)

We improved! We can still improve!

Minister of Education (2011-15) / Scholar (applied mathematics)

Diário de Notícias (7/12, p.11)

PISA 2015: quality and equity in education

Minister of Education (2005-09) /Scholar (sociologist)

Público

(7/12, p. 44)

A big round of applause for our schools

Journalist

Público (8/12, p. 48)

Justice for Nuno Crato [Minister of Education (2011-15)]

Columnist (politics and society)

Público online (8/12)

Parents and mothers of Portuguese students’ success

Journalist specializing in education

Expresso

(10/12, p.2)

Reality runs over the ministers of education

Journalist/Columnist

Expresso

(10/12, p.34)

What difference does it make?

Columnist and political commentator/ Scholar (Social and political sciences)

Expresso

(10/12, p.35)

That lost decade

Journalist/Columnist

Expresso

(10/12, p.35)

Our best results ... to date!

Minister of Education (2011-15) / Scholar (applied mathematics)

Público online (12/12)

School success has no owner

State Secretary for Education (current)

Público online (12/12)

PISA 2015, sloping results?

Member of parliament

Expresso online (12/12)

A matter of education

Columnist (economy)

We organise the content analysis according to two axes: how is PISA observed by the authors of articles about the PISA results; which are the main themes that emerge from the observations they produce about the PISA 2015 data. Two main interpretative lines emerge from our exploratory analysis: the reception of the 2015 data as a further moment for the consecration of PISA’s credibility; the use of the results, in this short period of reception, for processes of qualification and disqualification of educational policies and perspectives, directed both towards the past and the future.

Strengthening PISA status

Regarding the way PISA is observed by the article authors, two complementary elements are significant: the absence of objections to PISA13 and the further credibility ascribed to it. This consecration of PISA’s credibility occurs in various ways. It happens through the use of positive adjectives for PISA or through the naming of qualities that confirm its value: its great dimension, data richness, reputation, seriousness, expertise. It happens through the presentation of PISA data and results as a mirror of reality and through confident judgements of policies on the basis of the variation of results. And it also happens through categorical statements that confirm OECD views on PISA: it measures the quality of student learning in each generation; it drives countries to develop regular reflection practices; it breaks down ready-made ideas on education.

A final association is identified in the corpus: that which presents PISA as an element for the ordering of the political life in the education sector and for the governability of the education system. There is the imagination of the need to revise the national exams, as these are not showing the same progress in the results PISA does. And there is the appreciation of the existing consensus about the credibility of PISA assessment as a sign of a (desired) change in the way public policy is done in Portugal – more on the basis of evaluation and less on ideological prejudices. This understanding of PISA as a regulating device of other regulating devices and of the policies themselves places it on another level: as “third-party evaluator” in social processes of “surveillance and interpellation” of public authorities’ interventions (Cattonar & Mangez, 2014, p. 68).

The inscription of ‘performance interpretations’ in the policies trajectory

Portuguese students’ results were better than those of the previous cycle, they confirm a continuous trajectory of improvement and, for the first time, they are higher than the OECD average for the three literacy fields (Fig 2). In this scenario, it was the power of praise and hope, not that of guilt, which mostly emerged in the texts.14 Thus, the acknowledgement of progress is present in all. But only in few cases are the gains simultaneously attributed to overall improvements in performances and simultaneously to improvements of specific groups of students.

Figure 2 : Scores obtained by Portuguese students who were assessed by PISA, 2000-2015. Source, Marôco, 2016, p.9

The content analysis of the observations produced by the authors of the articles displays three recurrent themes, in which there are significant elements of tension: explanations for the observed improvement of results; qualification/disqualification of policies on the basis of the results; required policies on the basis of the results.

When explaining the progresses on the basis of PISA 2015 results the arguments concentrate on ‘to whom’ and/or ‘to what’ might the success be attributed (although with no mention of data or information from PISA 2015 reports, except in three cases). And they cross two distinct subject-matters: the extent and the combination of the considered explanatory factors; the time-span required to validate the existence of public policies effects upon the results.

In the first case, most texts tend to assign responsibility to a wide range of actors (and factors): teachers (work, competence and endurance in adverse circumstances); families (levels of parental education and expectations regarding school trajectories); ministers (competence); students (for their work). Only a few refer to a combination of factors, and those who do it assemble very disparate factors, rarely apprehended in a systemic manner.

But all the texts go on, and more profusely, about policies, which thus reinforces the perception that policies may make a difference. Most authors consider various public education policies introduced since the middle of the last century, but there is a debate on whether or not policies pursued by the last government have an effect on the results. It is a reaction present in several texts published after the previous minister [2011-2015] claimed an effect of his policies on the results (December 7). Some texts do so directly targeting the minister and counter-arguing, others recall the need to consider a wider time frame.

Still, although not referring to concrete interventions of the last government, several texts associate the improvement with the banner of its policies and with the discourse on the Portuguese school that it embodied in the previous decade: ‘exigency’. Consequently, the divide on whether or not to consider the most recent interventions is embedded in a broader one between those who defend educational perspectives that may be associated with the intervention of centre-left or centre-right governments. This divide is perceived at two interconnected levels: qualification/disqualification of policies; qualification/ disqualification of discourses (Table 5).

Table 5: Apologies and criticisms in the ‘observation’ of the results

Qualified

Disqualified

Public Policies

Extension of pre-school education

Priority intervention

Differentiation of pedagogical practices

Strengthening the quality of school spaces

Exams, curriculum standards, rankings

Horizontal stratification (vocational courses in basic education)

Horizontal stratification (vocational courses in basic education)

Discourses

The demand for results

The culture of ‘exigency’

Educationalists culture of complacency

Loss of quality of the education system

Inability of the school to promote social mobility

The (strictly) economic function of education

Combining the beliefs actors carry to the public action and the pragmatism of the political action (Barroso, 2009), the reception of PISA 2015 becomes a privileged space for the activation of scripts elaborated in the past and, also, in an exercise for the differentiation of views on education. However, it is also a space for the aggiornamento of orientations. This phenomenon is particularly evident in the way texts written by actors related to the centre-left almost entirely abandon the defence and the widespread use, mostly between 2005 and 2009, of pervasive evaluation policies (of students, teachers and schools). From this former attachment to evaluation little remains: the promotion of the external evaluation of schools is remembered in only one of the analysed texts; the invocation of ideas associated with the “policy evaluation” movement (see Pons, 2011) and the so-called evidence-based policy by actors connected to the university world, in line with the way in which they give credibility to PISA as a reference instrument for policies.

This withdrawal of the assessment policies from the discourse of the centre-left (and that of the current government [2015-….]) cohabits with a (new) agenda in which the problem of the (excessive) grade repetition is highlighted, a question identified long ago by OECD as a negative feature of the Portuguese system. Thus, some texts offer arguments for the fight against grade repetition, and in favour of diagnosis and early intervention in order to meet that end. Furthermore, in the ceremony for the presentation of PISA 2015 results, grade repetition was the subject reporters mostly captured, eventually due to the focus on the matter by those in charge of the management of PISA in Portugal and by the Minister of Education, for whom “PISA recommendations are mirrored in the government programme”15.

In other texts, the ‘grade repetition agenda’, backed by the mobilisation of analyses present in the 2015 reports and by the OECD “tutelary shade”, is opposed by calls for the preservation of standardised assessment policies, which were intensified in Portugal between 2011 and 2015, and for the discourse in favour of “exigency” and of a “focused and assessable” education, based on exams, as well as in favour of horizontal stratification with vocational courses in basic education. Some of them also claim to find in PISA support for such a solution.

Closing remarks

Over the past fifteen years, a reciprocal legitimating relationship between PISA and the policy actors was established and consolidated in Portugal.

There is a selective mobilisation of knowledge, in such a way that recurrent public policy questions are kept, come back or achieve greater visibility as educational and public agenda problems, according to the tendencies imposed by the results and by the political orientation of the contenders. If the credibility ascribed to the knowledge generated within PISA offers its national users added legitimacy for debating and proposing policy measures, such insistent use (even if superficial) in turn adds authority to the survey and to the scripts it transports. Consequently, it seems clear that also in Portugal the use of PISA credibility for the benefit of the ideas and policies to be put – or kept – in the policy agenda and to be implemented is, somehow, superimposed to the expected rationalisation of public action. This observation supports the hypothesis that “knowledge for learning” processes are not often considered when the actors mobilise PISA data to elaborate arguments about their systems (Pons, 2012). In any case, in face of the existing studies, the question of what politicians (and, more generally, public policy in Portugal) have learned from the use of the ‘mirror’ remains open.

The analysis of PISA trajectory in the policy processes in Portugal discloses what the literature on knowledge-policy relationships has, since long, been exposing: the impact of research knowledge on policies depends on a combination of elements, like the level of the policy actors’ knowledge and expertise on the subject, or the compatibility between that knowledge and actors’ ideological compromise or strong interests (Weiss, 1990). It also seems clear that the critical factors for the presence of comparative assessment knowledge in policies relate to the origin and type of knowledge, to the ways in which knowledge is mobilised and, likewise, to the contextual and interest-related features of political action and to the historicity of the rules and beliefs that guide it. In this context, the emergence in Portugal of new actors and ways of translating PISA data/information/knowledge for policy is a trace of originality in the processes of reception that requires further analytical observation.

In sum, PISA data and analysis open and close routes and establish a set of possibilities for policies. Mobilised in various sociocultural spaces, PISA is acted upon by systems of meaning and power relations, which it also helps rewriting, in that way producing specific effects (Popkewitz, 2003). Thus, the regulatory role of PISA seems clear: it enables the emission of judgments about the school systems, the construction of opinion on policy decisions, the reasoning of decisions based on knowledge perceived as ‘real’. Of course, this does not come out from the evaluation per se. The politics of reception are an important part of this, too.

References

Afonso, N., & Costa, E. (2011). A circulação e o uso do PISA em Portugal. In L. M. Carvalho (Org.), O espelho do perito (pp. 107-125). V. N. Gaia: Fundação Manuel Leão.

Barroso, J. (2009). A utilização do conhecimento em política. Educação & Sociedades, 30(109), 987-1007.

Carvalho, H., Ávila, P., Nico, M., & Pacheco, P. (2011). As competências dos alunos. Resultados do PISA 2009 em Portugal. Lisboa: CIES-IUL.

Carvalho, L. M. (2014). The attraction of mutual-surveillance of performances. In T. Fenwick, J. Ozga, & E. Mangez (Eds.), World yearbook of education 2014: Governing knowledge (pp. 58-72). Oxford: Routledge.

Carvalho, L. M., & Costa, E. (2016). The praise of mutual-surveillance in Europe. In R. Normand, & J.-L. Derouet (Eds.), A European politics of education? (pp. 53-72). London: Routledge.

Carvalho, L. M., & Costa, E. (2009) Production of OECD’s PISA: Final report. Project KNOWandPOL. http://www.knowandpol.eu/

Cattonar, B., & Mangez, E. (2014). Codages et recodages de la réalité scolaire. Revue Internationale d’Education de Sèvre, 66, 61-70.

CNE (2010). Impacto das Avaliações Internacionais nos Sistemas Educativos. Lisboa: CNE.

CNE (2015). Investigação em Educação e os Resultados do PISA. Lisboa: CNE.

Costa, E. (2011). O PISA como instrumento de regulação das políticas educativas. Tese de Doutoramento. Universidade de Lisboa, Instituto de Educação.

Delvaux, B. (2009). Qual é o papel do conhecimento na acção pública? Educação & Sociedade, 30(109), 959-985.

Fernandes, D. (2014). Avaliações externas e melhoria das aprendizagens dos alunos, In CNE, O Estado da Educação 2014. Lisboa: CNE.

GAVE (2001). Resultados do Estudo Internacional PISA 2000. Lisboa: ME/GAVE.

GAVE (2004). Resultados do Estudo Internacional PISA 2003. Lisboa: ME/GAVE.

GAVE (2007). PISA 2006 – Competências científicas dos alunos portugueses. Lisboa: GAVE.

GAVE (2010). PISA 2009 - Competências dos alunos portugueses: Síntese dos resultados. Lisboa: Lisboa: GAVE

Justino, D. (2015). Abertura. In CNE, Investigação em Educação e os Resultados do PISA (3-17). Lisboa: CNE.

Marôco, J. (Coord.) et al. (2016). PISA 2015 – Portugal. Volume 1. Lisboa: IAVE.

MEC (2015). Educação Pré-Escolar, Ensino Básico e Secundário. Relações Internacionais. 2011-2015. Lisboa: MEC.

Ozga, J. (2012). Knowledge stocks and flows: Data and education governance. In T. Fenwick, & L. Farrell (Eds.), Knowledge mobilization and educational research. London: Routledge.

Pons, X. (2011). Policy evaluation in education in France. Educação, Sociedade & Culturas, 34, 35-50.

Pons, X. (2012). Going beyond the ‘PISA shock’ discourse: an analysis of the cognitive reception of PISA in six European countries, 2001-2008. European Educational Research Journal, 11(2), 206-226.

Popkewitz, T. (2003). National imaginaries, the indigenous-foreigner, and power. In J. Schriewer (Ed.), Discourse Formation in Comparative Education (2nd. ed., 261-294]. Frankfürt: Peter Lang.

ProjAVI (2013). PISA 2012, Portugal - Primeiros resultados. Lisboa: ProjAVI.

Steiner-Khamsi, G. (2004). The global politics of educational borrowing and lending. New York: Teachers College Press.

Van Zanten, & Pons (2006). Knowledge circulation, regulation and governance. In Aa.Vv. KNOWandPOl Literature Review, Vol I. http://www.knowandpol.eu/

Weiss, C. (1990). The uneasy partnership endures: social science and government. In S. Brooks, & A.-G. Gagnon (Eds.), Social scientists, Policy, and the State. NY: Praeger.

1 Texto original: Carvalho, L. M., Costa, E., & Gonçalves, C. (2017). Fifteen years looking at the mirror: On the presence of PISA in education policy processes (Portugal, 2000-2016) European Journal of Education, 52 (2), 154-166.

2 https://www.youtube.com/watch?v=M0zY8IcEIT8. Accessed 22/11/2016

3 Series aim is to “[highlight] initiatives taken by education authorities around the world to help school students do better” https://www.oecd.org/pisa/pisaproducts/videoseriesstrongperformersandsuccessfulreformersineducation.htm.%20Accessed%2022/11/2016

4 https://all4ed.org/debunking-seven-myths-about-pisa/ Accessed 22/11/2016

5 www.governo.gov.pt. Accessed 19/12/2007

6 The CNE is an education policy consultancy body of the Ministry of Education (ME).

7 See note 5.

8 Aqueduct, in English.

9 The philanthropic foundation, Fundação Francisco Manuel dos Santos, was created by a Portuguese economic group, and publishes, funds research, promotes discussions on various social issues, working with ‘experts’ and producing data on Portugal and Europe https://www.ffms.pt/en/about-us/2mission-and-goals. Accessed 02/12/2016

10 http://www.aqeduto.pt/apresentacao/ Accessed 02/12/2016

11 In an earlier phase of the project, the team also included one other member that worked previously (2008-2011) as NPM at the GAVE.

12 The presentation by the President of CNE on a seminar on PISA results is an important source of our exploratory analysis (Justino, 2015).

13 There are some considerations on methodological questions (and on a possible exclusion of students who follow a vocational path), but the authors judge reproach national policies and do not blame the survey. Such possible exclusion was actually later denied by the National Programme Manager, in 12/12.

14 One text questions if the improvement might result from horizontal stratification policies, which the author associates to a possible exclusion of students from the assessment. See previous note.

15 Expresso online [Weekly newspaper]. Accessed: http://expresso.sapo.pt/sociedade/2016-12-06

Ficha Técnica

Título: PISA – Programme for International Student Assessment: Um Instrumento de Regulação da Educação. Compilação de Estudos (2008-2017)

Organizador: Luís Miguel Carvalho

Edição: Instituto de Educação da Universidade de Lisboa

1.ª edição: Dezembro de 2017

Coleção: Estudos e Ensaios

Composição e arranjo gráfico: Sérgio Pires

Disponível em: www.ie.ulisboa.pt

Copyright: Instituto de Educação da Universidade de Lisboa

ISBN: 978-989-8753-40-3

Este livro é financiado por fundos nacionais através da FCT — Fundação para a Ciência e a Tecnologia, no âmbito da UIDEF — UID/CED/04107/2016.