Academic research and policy research are two different things

The case for investing in context-specific policy research and implementation strategies

Thank you for being a regular reader of An Africanist Perspective. If you haven’t done so yet, please hit subscribe in order to receive timely updates.

I: How best can we harvest the fruits of the credibility revolution?

Alex Tabarrok of George Mason University once raised an interesting question regarding the field of development economics:

I used to think there was such a thing as development economics. There are still richer and poorer countries, of course, but is there a “development economics,” a special type of economics for poor countries? I don’t think so.

The same question can be asked of policy research in low-income countries — which is the subject of this post.

---

But first, some throat clearing. This post is not an installment in the RCT wars. Neither is it meant to speak to broader methodological fights in the empirical social sciences. Instead, it is an exploration of how we can best reap the fruits of the credibility revolution (with regard to both qualitative and quantitative works) to improve both the study and practice of policymaking in low-income states. The target of my critique isn’t rigor, but how we have come to use the output so produced. I hold the view that a reasonable reaction to perceived overrating of RCTs by some (as the only credible way of knowing the world) shouldn’t be to demand for fewer RCTs, but to conduct more studies of all kinds in more countries/contexts, period. A lot of the world remains terribly understudied.

This post’s primary intended audience are government policymakers as well as policy researchers in think tanks, philanthropies, and high-income countries’ international development agencies.

---

The credibility revolution unambiguously improved research in the social sciences. However, when it comes to policymaking, a yawning gap has developed between rigorous academic social scientific research and policymaking in many (if not most) developing countries. This is because (i) policymakers have erroneously come to view policy research as indistinguishable from academic research; and (ii) policymaking and implementation have largely been reduced to technical programmatic endeavors, and disembedded from contextual economics and politics.

Let’s be blunt. Regardless of pedigree, academic researchers are often poor policymakers. This is simply because academics are neither trained nor motivated to be good policymakers. If anything, unless we are deliberately widely read and curious about the societies we study, our narrow training typically leaves us insufficiently appreciative of the complexities and rigors of policymaking and implementation. Furthermore, systems thinking — which is absolutely critical for succeeding at policymaking — isn’t our forte. We are trained to be narrowly focused, a fact that is reinforced by the incentive schemes that govern our promotion and recognition in our different fields.

The principal objectives of what we do as researchers are to win grants, publish papers/books, and please our peers — so we can get promotions and other accolades. And despite all our heroic claims about policy relevance, what we do to please fellow academic reviewers or granting outfits is not always useful for policymaking in the real world. To make matters worse, there are strong disincentives against studying the same policy in the same context over time. For example, what gets rewarded is either being a global education expert that can at once speak to Malawi, Peru, and Cambodia; or being an empiricist who hops from studying cash transfers in Bolivia to agricultural subsidies in Benin to teacher training in Bihar. These approaches are great for publication and careerism in academia and/or International Development. But they are ill-suited for long-term policy research, formulation, and implementation in low-income countries (or anywhere, really).

On the other side of this problem lies, for lack of a better term, highly impressionable government policymakers and program managers in leading donor and philanthropic organizations. The need to sound smart by claiming to be champions of “evidence-based policymaking” (who’d be against that?) has resulted in these important actors abdicating their duties. Many have fallen to the temptation to hide behind academic rigor and implied expertise, even as they avoid doing the hard work of bridging the gap between academic research and actual policymaking and implementation. This state of affairs has created four major problems:

1. By abdicating their duties and outsourcing important parts of their roles to academics, policymakers and development program managers have let their policymaking skills atrophy (or never developed them at all — have you seen the syllabi and faculty at profiles at leading international development policy schools?)

2. While government officials in low-income countries, donors, and philanthropists may still be in positions to set the agenda (often in a faddist fashion), they’ve nonetheless limited their potential for impact by not fully understanding academics’ incentives as part of the evidence-based policymaking enterprise (which reinforces faddist approaches to policymaking). Policy directions in major sectors like agriculture, education, and healthcare have since become dominated by global expertise and rendered technical problems that are, for all practical purposes, been divorced from domestic political policymaking processes.

3. The abdication of policymaking to academics has led to a “tyranny of experts” that is at once anti-politics and anti-economics. Claims to expertise have been used to sidestep political institutions. Standardized global approaches have made policy recommendations far removed from local economies’ ability to support them; not to mention the entrenchment of a paradigm that views development work as being about implementing anti-poverty programs instead of making policies to unleash commercial revolutions and broad-based economic transformation.

4. The entire enterprise of knowledge production for policymaking has come to exist within a paradigm that conceptualizes the problem of under-development to be principally driven by ignorance (if only we knew whether it’s good to feed kids!) and poor policy implementation (largely due to “lack of political will”). Never mind that most of the policies being peddled tend to be poorly conceptualized to begin with. Never mind interest groups and distributive politics. Never mind trying to figure out how to squeeze good learning outcomes within a country’s budget envelope. Never mind figuring out how countries should manage their macro policy.

These are problems that can only be solved by policymakers (including program managers working for donors, IFIs, and philanthropies) reclaiming their roles as bridges between rigorous academic research and policy formulation and implementation. That means internalizing the fact that the craft of policymaking is both an art and a science. Socio-historical context matters. Economic incentives matter. Messy distributive politics matter. Institutions matter. The strength of policy transmission mechanisms matters. And finally, success comes not from depoliticizing policymaking or pretending that development can only be understood from the perspective of household-level dynamics, but by anchoring the entire policy cycle in domestic economics, politics, and institutions, warts and all.

A capacity building workshop for subnational bureaucrats organized by the Kenya Institute for Public Policy Research and Analysis (KIPPRA). Source: KIPPRA

To reiterate, hiding behind academics’ rigorous research won’t cut it. Merely being a developing country trying to do catch-up growth/development doesn’t suspend the standard rules governing the political economy of policymaking. In fact, such contexts require greater attention to economic and political incentives in context (as opposed to assuming the ability to technocratically tweak variables on a dashboard). Policymakers often have to build the policy formulation and transmission mechanisms under less-than-ideal political and economic conditions in the process of policy implementation.

With all this in mind, the best way to harvest the fruits of the credibility revolution is to build systems of knowledge production about policy (i.e., actual policy research) that are as close as possible to policy implementation. Ideally, this means re-empowering official government policymakers and processes and investing in country-based or regional think tanks and university research institutes in different sectors. The ultimate goal should be to cultivate country-based cadres of policymakers who understand the economics and politics of policymaking (i.e., appreciate the contextual variables highlighted above) while also being conversant with the good research being produced by academics. Currently, the false promise of high scientism abstracted from context leads way too many smart people to devalue the cardinal importance of intimate knowledge of context and involvement of stakeholders to successful policymaking and implementation.

II: Doing policy research well

As academics, we should be free to do whatever research we want using whatever methodologies we want (ideally, though, the type of question should drive the choice of methodological approach). Policy research, on the other hand, should pay close attention to context (obviously, this does not imply ignoring the benefits of comparative analyses). Importantly, policy researchers should understand that the fundamental challenge facing low-income countries engaged in catch-up development is rarely what to do, but rather how to do.

Notice that this is diametrically opposed to the (overplayed) radical ignorance that motivates most of purely academic research that is deemed to be policy relevant. For policymakers, being evidence-based doesn’t mean one should pretend to be ignorant ex-ante about whether it is a good idea to feed or immunize children, have a job or disposable income, expand both the capacity and quality of teacher training, or build a dam. Furthermore, policymakers ought to understand that people can be mobilized to collectively achieve goals that may ex-ante appear impossible (this is a terribly under-appreciated input in the development process).

The point here is that commitment to evidence-based policymaking should not be used as an excuse to waste time and resources figuring out what to do even when it is obvious unwarranted. Policymakers should lean into the advantage of doing catch-up growth by carefully learning from others, domesticating proven interventions, and, yes, experimenting/piloting interventions where necessary.

While it is fine to solicit inputs from academics, policymakers must understand such researchers’ limitations. Beyond the specific career incentives outlined above, it is understandable why academics are predisposed to a posture of radical ignorance. Since many study interventions in countries other than their home countries, they are often objectively ignorant about context (as anyone who has fielded a “let me pick your brain” meeting would agree). In addition, as outsiders committed to (mostly) doing no harm and who might feel a little icky about potentially failed interventions in low-income countries (and perhaps self-aware of their lopsided influence on impressionable government ministers and donors), it is obvious why their first impulse is always to try and figure out what to do, as opposed to paying attention to the problem of how to do. Furthermore, the need to cater to funders often leads to an over-emphasis on particular kinds of questions, evidence, and/or funders’ budget constraints as a reason to figure out what to prioritize/do (as opposed to viewing this as a political prioritization problem that should be channeled through local institutions and fiscal capacities).

Combined, all these factors reinforce the preponderance of incrementalist apolitical/technocratic tweaking and household/individual-centric interventions as the smart way to make development policy, rather than valiant quests towards broad-based macroeconomic transformation.

Policy researchers and officials in low-income countries should completely avoid these self-imposed cognitive and operational shackles at all costs.

The business of policymaking in low-income countries should squarely focus on broad-based economic transformation. To channel Tabarrok, low-income economies should be treated as “normal.” That means investing in contextual knowledge as a matter of necessity; designing policies that are aligned with local fiscal and implementation capacities; leaning into local institutions and distributive politics to anchor policies; and embracing long-term thinking rather than being driven by faddist magical thinking.

In practice, this would mean policy research agendas with the following features:

1. Prioritization of how to do questions and the building of supporting policy transmission mechanisms within specific contextual systems. This orientation should also be reflected in the training of policy researchers and bureaucrats.

2. Explicitly incorporating a contextualized political economy approach. Instead of imagining a do-gooder policy planner out to implement first-best solutions abstracted from context, policy research should strive to make nth-best recommendations that are robust to governments’ fiscal and administrative-bureaucratic capacities as well as the realities of local distributive politics.

3. Appreciate the virtues of learning by doing. Policy cycles stretch far into the future, driven by iterative processes of learning and adaptation. Implementation and learning from the same in order to ensure sustainability are often the hardest part. This means that good policy research is likely to be done by people who are incentivized to have a long-term interest in policy success and understand that the iterative nature of implementation.

4. Be anchored as close as possible to implementation. There’s a sociology to knowledge production and uptake. Good policies are likely to emerge if researchers, bureaucrats, politicians, and beneficiaries belong to the same societies and epistemic communities. This is most likely to obtain if policy research is conducted by local (or sufficiently localized actors) that can actively and passively absorb contextual knowledge and leverage the same in influencing the process of policy formulation and implementation.

5. Explicitly stand separated from academic research (while embracing methodological diversity and rigor). The goal should not be to dumb down policy research, but simply to acknowledge that policy researchers are optimizing different things than academic researchers (including academics who research policy). This means, for example, that policymakers can still run experiments or pilot policies and learn from them; and conduct comparative case study analyses, etc. However, the difference would be that policymakers would singularly focus on contextual learning, rather than necessarily maximizing legibility of findings to journal reviewers or far-removed donors/funders.

Finally, it is worth emphasizing the importance of leaning into local political processes. Instead of viewing the messy distributive politics of policymaking as a nuisance, policy researchers should embrace it. Open politicking over policy enables policy learning among bureaucrats and voters alike; cultivates interest groups that can support policy implementation; creates opportunities for compensating would-be losers; and increases the likelihood that adopted policies will be reinforced by other contextual policies or system-wide variables.

For these reasons, policy research ought to be as close as possible to context and oriented towards practice in order to ensure that critical features of policymaking and implantation are baked into the knowledge production process. It’s only through such an arrangement that low-income countries would be able to adopt and domesticate lessons from elsewhere (like, for example, China and Japan did) and not end up in cycles of reforms with little to show for it.

III: Conclusion:

Not enough people appreciate how bad policymaking can be in low-income countries (and no, it’s not just because of corruption or bad governance). For example, I often hear people say that Kenya formulates good policies, but implementation is the problem. This is false. The fact of the matter is that, with a few exceptions, public policymaking in Kenya is atrociously bad. Except perhaps during the Mwai Kibaki era (2003-2013), the quality of policymaking has steadily declined since the peak glory days (late 1960s - mid 1970s) when people like Charles Karanja were in charge and absorbed in the business of making smart transformative policies.

I cannot stress this enough: People like Karanja firmly understood the assignment of designing and implementing transformative developmentalist policies. Their successors lost the plot.

After the 1970s policymaking atrophied under the combined weight of economic decline, increasing dependence on helicopter foreign expertise (notice that the early postcolonial foreign expertise was fairly domesticated), and the decoupling of “apolitical” policymaking from intra-elite dealmaking and interest group politics. Importantly, the decoupling of policymaking from dealmaking created the illusion that policymaking is merely technocratic and can be shielded from economic realities and messy distributive politics. It shouldn’t surprise anyone that most of the “good” apolitical policies have rarely survived contact with the realities of dealmaking and distributive politics at the implementation phase. A good illustration is Kenya’s ongoing eduction reform effort, the subject of a future working paper and post. The reforms (see here and here) took a functioning system that needed reforms and completely destroyed it (with inputs from donors and researchers, of course).

The only way out of this problem would be for policymakers to get back to being the bridge between research and practice in context; and not wait to be rescued by off-the-shelf solutions cobbled together by academic researchers.

Going back to the quote from Tabarrok above, there are two ways to interpret the motivation behind his question. It might be that people tend to suspend the standard rules of economics and politics when studying policy in low-income countries. Alternatively, it could be that the presence of similar policy problems across countries at all income levels obviates the need to condition approaches to policy questions on levels of income. Tabarrok sounds sympathetic to the latter interpretation.

I would be more sympathetic to the first interpretation. By virtue of having the “advantage” of mostly engaging in catch-up growth, the policy choices and how to implement them (especially in domains that require technology adoption) are necessarily different in low-income countries relative to their high-income counterparts. This distinction does not disappear simply because similar policy challenges might exist in certain pockets of high-income economies. There are good reasons to approach policy research differently depending on a country’s income level. Critical policy transmission mechanisms tend to be weak or non-existent in such countries. There’s often not enough money to fully implement needed policies at scale. The combination of administrative-bureaucratic weakness and resource constraints mean that systems tend to be out of practice, with outcomes disproportionately dependent on personalities involved.

However, the fact that one does policymaking/research in a low-income country with these characteristics is not a reason to suspend the basic rules of economics and politics. Economic growth and development in low-income still require good-enough institutions, protection of property rights, state capacity for contract enforcement, working markets, investments in human capital, large firms that generate mass employment, balanced distributive politics, and the like. These are excellent reasons to ensure that policy research, formulation, and implementation are as embedded in local political economies as possible. Which is to say that Tabarrok is kind of right: make policy research, formulation, and implementation in low-income countries normal again.