Disclaimer: This is not a formal paper of such yet all references are hyperlinked with the resource in case you want to dig deeper.

Evidence-based policy making (EBPM). Ah, this buzzword has its own perks, hasn’t it? Reminding ourselves that policy decision used to be lack of data and such. “We should use sufficient data so then we could make better decision” or so it says, “Lack of accuracy and evidence by research contributes to policy failure”. More over, I have found many papers or research reports about Indonesia with title “……. : Evidence from Indonesia”.

But then is it really? To what extent lack of evidence contributes to policy failure? If you already have the evidence, again to what extent, it would make the decision better? If the demand of evidence is already there even become mandatory, how it could influence the policy-making process? What about the politicisation of data and research, does it make the evidence become not exclusively in a vacuum as it depicted?

Question question question. And those questions actually lead me to brief writing last year. Of course, I did not make a comprehensive and rigorous study, yet this is more like an effort to compile some ideas that I hope could trigger another question at the end.

The Evidence Tug-and-War

‘Evidence’ is already longed-known as the basis argument for policy since Cold War. In the middle of science battle with Uni Soviet in the late 1950s, US government marked the rise of scientific approach in policy formulation which then increases the role of social scientists and researchers; policy expertise became a growth industry for think tanks, university research institutes, and management consulting firms (Fischer, 2003). Even though the term ‘evidence’ previously was not literally mentioned, there was superiority on policy science. The axial principle — the source of innovation and policy formulation for the society centralized and organized around knowledge (Bell, 1973) — existed to answer the need for better public policy during Cold War.

Another stepping stone for EBPM initiative was in the late 1990s when UK Government under Labor party wrote White Paper Modernising Government. It makes clear that policy decisions should be based on sound evidence and to make policy in the philosophical mind of “what works” (Nutley & Webb, 2000). Evidence-based policy’s mission is to choose an intervention on the basis that it has a reasonable chance of repeating successful outcomes achieved elsewhere (Pawson, 2006, p. 22). Public policy is a matter of options and evidence is necessarily used to make the right, rational choice. This is the brief notion of rationality approach towards public policy: a mission to rescuing public policy from the irrationalities and indignities of politics (Stone in Wesselink, Colebatch, & Pearce, 2014).

However, the practice of EBPM is not an easy game. Policy-making itself contains a very complex, complicated, and intertwined linkage between actors, information, institution, and society itself. Social problems are viewed through particular spectacles with a different and limited amount of time as well as the cognitive capacity to comprehend the problems. Evidence, then, is believed can overcome the limitations of government and scientists can help policymakers to solve the problem (Cairney, 2016).

Generally, there is blinded hopeful on EBPM, which implies a naïve view that there is (and should be) a direct and unproblematic link between scientific evidence, policy decisions and outcomes (Cairney, 2016). Yet, in fact, most of the policy making process is unpredictable and not linear; the causality relation between evidence and outcomes of public policy is irregular and intervention works selectively (Pawson, 2006: 22–3). Policy-maker tends to have constraints to make the most optimized public choices and this limitation is what we have known as bounded rationality.

Bounded rationality theory is an alternative theory proposed to answer challenges on rationality debate on decision-making. It elaborates the limits of rationality in various ways including incomplete information about alternatives (Simon, 1972, p. 163). The ‘incomplete information’ here is not about the quantity and quality of provided facts yet also how decision-maker process the information. One of the significant attributes is that perception of information is selective and perception itself is not independent since particular environment and other mental factors construct (Bendor, 2010, p. 16). Cognitive capacity is limited when policy maker faces complex and difficult problem yet it is not irrationality (Selten, 1999).

Then the question evolves, what kind of rationality policy-makers use to analyse problems and eventually reach policy ends? Winship (2006) provides another concept of “puzzling about policy” to explain that rationality used in policy-making is not about choosing among a set of possible options. It is about trying to figure out how it might be possible to “fit the pieces of their puzzle”. The most important ability here is not good evidence, but good perception. It is impossible to separate values and interpretation, to add, policy-makers also live in the realities of the political world. Evidence can be used to provide more information and minimize uncertainty but it cannot solve ambiguity (Cairney, 2016).

On a more practical world of policy-making, indeed there is nothing particularly controversial about using evidence as the basis of policy, yet the limits should be recognized. First, there is time scale disparity between evidence and policy-making process. Choi et al. (2005) argue that the whole process of science is very slow since it builds on research findings. Scientific studies and evidence need careful, long term advance planning and preparation. On the other hand, policy-makers cannot afford long-time spans attention since each one has to cover vast thematic fields with chronic time pressure (Cable, 2004). There is communication gap between policy-makers and researchers in such way thus evidence has a tough role-play to gain wider credibility amongst decision makers (Sutcliffe & Court, 2005)

Second, attached political culture in policy-making cannot be ignored even though the evidence has shown. Trying to influence each other’s thinking and behavior through a conversation on the basis of evidence or argument is common in the policy-making arena (Susskind, 2006). The evidence here is not the most fundamental foundation of policy ends, but an instrument to convince and change another’s mind. In addition, policy studies have been long known agenda setting and democratic theory, but what sometimes being ignored is the possibility that some individual or institution may hold exclusive power over the agenda, in other words: ‘agenda control’ (Majone, 2006). The plausible existence of agenda control on policy-making infers complex relations between policy-makers and evidence on the particular issue. ‘Evidence’ is prone to politicization through attempts on realigning and re-regulate contested practices of policy-relevant knowledge production (Strassheim & Ketunen, 2014).

On the other side, the problem of EBPM also lies on the limitation on evidence production (supply side). The limitation of production is not merely on limited resource management but researchers and scientists face difficulties of validity as well (Solesbury, 2001). First is the nature of social explanation. Pawson (2006) argues that there exists ‘open system’ where social systems are transformative and prone to contextual changes whereas then social research act itself is very dynamic and always tend to disturb what it is trying to describe. This is what makes some cases are more difficult to identify by researchers and scientists especially cases that have long tangled historical story and overlapping causes/effects. Undoubtedly there are issues where the evidence is relatively clear (for example on medical and environment issues) yet it is harder to find evidence-based solutions to more complex and nuanced cases (for example the identification of inequalities) (Cairney, 2016). Thus, the supply of evidence for policy is contested.

Second, among scientists themselves, not all favor the same hierarchy of evidence (Cairney, 2016; Choi et al., 2005; Pawson, 2006). Moreover, hierarchy-based on study design is somehow problematic since they are often become too narrow, under-rated good observational studies, lose useful information, pay insufficient attention to program theory, and provide an insufficient basis for recommendations (Nutley, Powell, & Davies, 2013). What counts as good evidence is still debatable and further clarification in tailoring the evidence standard is needed.

Those push-and-pull forces between supply and demand are what makes evidence than is even theoretically really difficult to be determined and measured by ownership. The tangled, spider-web relations between actor-actor, actor-evidence, and evidence-evidence are highly prone to create confusion. On the other hand, there is tendency of institutional-ego, from both sides, to show themselves capable of answering social problems.

So, what?

From what I understand, in the case of Indonesia, there are increased numbers of independent research institutions/think tanks outside the universities and government bodies. Public policy is forcefully challenged and open for direct assessment. The ‘evidence’ problem essentially on the supply side has been acknowledged through various approaches including the solutions such as resource and availability of government-provided data (Bachtiar, 2011), research capacity in higher education institution (Rakhmani & Siregar, 2016), and the human capital/organizational problem in knowledge sector (AUSAID, 2011). However, in the context of research conducted by Indonesian civil society organizations (CSOs)/think tanks, it remains unclear on the impact of their research activities, not to mention that they are heavily reliant on foreign donor funding and highly prioritized to consulting work (Hadiz & Dhakidae, 2005).

I’m not sure if there is any comprehensive mapping on the relations of Indonesian think tanks/non-government research institution with the government. A study on Indonesian policy-making also notes that most of its respondents (MPs and civil servants) mentioned very few CSO research centers which may indicate weak links between such organizations and policy-makers (Datta et al., 2011). It mentions that the weak link is major exceptional for SMERU Institute since most director-generals and directors in Ministry of National Development Planning (Bappenas) have good relations with senior researchers from SMERU. However, the issue of independence on financial and agenda setting, strategic communication, and engagement with stakeholders are raised.

Then?

It is still become a puzzle for me since I might detach from the real practice of think tanks or evidence demand/supply chain in general. Yet questioning this whole EBPM reassures me that production of knowledge is never that easy. Especially with more phenomenon in this (so-called) post-truth era, gaining public trust through EBPM could be detrimental (potentially). Especially when most citizens perceive experts and scientist as part of elites who subjected them into numbers and statistics.

Policy experts, consultants, and scientists could be very confused with pure morality-based policy, for example. Where’s the fact that banning out-of-marriage sexual activity could lead to better society? There’s none. But in this context, the fact is not relevant if most policy makers are pushed by some interest groups, political affiliation, or even perhaps by their own standard of morality and personal values.

Should we provide the hard fact then? Does it work if the ambiguity of the topic still could not be minimized? Or maybe, just maybe, we should think another way to negotiate and lobbying so it could not even pass the ratification process?

Question question question.