At their September 28 meeting, the US Advisory Commission on Public Diplomacy announced a report they had commissioned from a research team at UT Austin. Its subject of evaluation and measurement for public diplomacy is undoubtedly important and a significant priority for governments engaged in public diplomacy around the world. This report owes its existence to the efforts of the Advisory Commission’s former Executive Director, David Firestein – an intelligent and articulate advocate for public diplomacy concerns. Frankly it’s surprising that it took this long for the Commission to get to this subject, but it’s a significant step in the right direction. The report itself, however, is not perfect.
Matthew Armstrong wasted no time in offering a thorough and at times stinging assessment of the report. Having read both the report and Armstrong’s commentary, I’ve come to a few conclusions:
1. The report is a commendable and surprisingly systematic attempt to devise a flexible evaluation tool.
2. It’s not ground-breaking in its recommendations – but then again I don’t think it was intended to be. It’s designed to provide a tool for policy evaluators to consider programs based on previous experiences.
3. I think Armstrong’s negative comments about the project’s formative research expressed a frustration many public diplomacy watchers share about previous evaluation work.
4. I also think Armstrong’s critique: that the report offers nothing new in terms of criticism of US PD is fine, but that wasn’t the point of the report.
5. The report authors were not able to interview and survey enough people to do a thorough analysis of US strategic thinking and culture about public diplomacy. Then again, I’m not sure they needed to. See above.
6. The PD-MAP is a neat tool. But it’s strangely idiosyncratic and at times arbitrary in its recommendations for how to measure outcomes. It’s not well cited – which is important when you consider all the different dynamics they are trying to measure.
7. I’m starting to appreciate John Brown’s position more on public diplomacy research (!). See below.
I won’t repeat the summary work done by Armstrong in his lengthy treatment of the report. I admit I was very skeptical of the report before reading it. There has been a lot of studies of US public diplomacy, commissioned by the GAO and others, that measure what the government is doing and not the result. Effectiveness is often inferred from what the report authors call “outputs.” The report, however, is very much interested in “outcomes.” Connecting outputs to outcomes is not an easy task – it requires diligent research, analysis, and indeed access to data that studies can use to make claims about outcomes. And here’s the kicker – it also requires theories to guide analysis, because we’ve developed pretty good ideas for how and why things happen that can be explained from data analysis.
For example, if you’re going to investigate persuasion effects of, say, selected exposure to a particular message, visual, or experience – then you don’t need a theory of public diplomacy. You need to employ established and reliable measures of effect developed in relevant fields like social psychology, communication studies, and public relations/strategic communication. Too bad the real world of public diplomacy doesn’t allow much for field work, quasi-experimental, or controlled experiments on persuasion effects. But I digress. We get it. Measuring public diplomacy is hard.
There’s a reason why the academic study of international relations and foreign policy in general has very few durable theories and hypotheses that have held up across contexts. A predictive, elegant model that is useful for international affairs is a rare thing to come by. The most sophisticated quantitative analyses very often deal with very specific parameters, situations, and data. The messy particulars of different embassies, cultures, operational contexts, policy programs, policy objectives, and research subjects makes a comprehensive model for PD seem almost fantastically absurd in prospect. So what do we get with the UT Austin report?
The report did not set out to develop a predictive model, let alone a theory, of public diplomacy, though it does claim to measure effectiveness against expectations established systematically from flexible factors and priorities. In my opinion this is a good thing. Rather, it’s pretty clear the group devised a way to catalog, organize, and code program information in such a way as to allow policy programmers to get a real picture of how effective their programs have been from a strategic perspective.
The three main strategic goals of PD the authors derived from their research and used for evaluation are:
Increased Understanding of the US
Increasing Favorability toward the US
Increasing US Influence of the US.
So, the first part of the report deals with how the authors settled on these three primary strategic objectives for PD. The second part builds on the objectives, to illustrate how you might code and standardize data in such a way as to model effectiveness against these objectives. Their derived tool, the “PD-MAP” model, is a flexible system to accomplish analysis of effectiveness (linking outcomes to outputs) that is apparently based upon a Multi-Criteria Decision-Making Model. Admittedly, I don’t know much about this concept, nor the theories and studies that have been used to develop the concept. But the benefit, the authors argue, is that you can tweak the model in any way you like to account for the relative importance of the objective, the risks involved, the expectations, etc. Through data standardization, the model combines results of different effectiveness measures into a relatively straightforward depiction.
I’ll cut right to the chase. The proposed PD-MAP evaluation tool is a great way to organize and compile evidence of effectiveness in such a way as to measure outcomes by standardizing results from a wide-ranging series of pre and post hoc measurement data. Yet the overall system relies upon a set of assumptions that makes me pause, because the real heavy-lifting of data collection and analysis aren’t that specified. What we have instead is a policy and strategy management tool.
Remember the saying about “when you assume…?”
The ways in which the authors describe sweeping subjects like influence and understanding are arbitrary. Yes, they do offer descriptions, but little justification by previous thinking on what those concepts signify as sets of processes. They just offer a definition and go from there, with effectively no citation or reference to why they made those decisions, nor why the definitions they choose are pertinent to public diplomacy. For example, there is a subsection on “favorability dimensions” – where did they get this stuff? I get it that we need relatively stable operational assumptions for these concepts to start measuring them – but I still find the conceptualization rather thin – especially when you consider the different sites of inquiry they identify (elites, governments, everyone else, etc.). They get around this problem by ultimately leaving much of the definitional work up to the person on the ground doing the analysis (assigning weights, objectives, etc.) who might better understand the particular situation. But this flexibility, in my mind at least, starts to erode the utility of the tool – especially if the significance of the variables (can we call them that?) are so subjective to the person who would use the model.
But yet another problem with these concepts is that they may be difficult to separate analytically when considering causation. The authors do admit to this problem in various sections of the paper, but it’s still present. Put simply, “increasing understanding of the US” could very well lead to “increasing favorability” in ways that are not obvious from the suggested measures they provide. Disentangling these concepts looks great on a policy checklist, but much murkier when making knowledge claims. Which leads me to my next quibble, the measurement methods discussed within PD-MAP.
The authors offer a wide array of suggested measures via example, supposedly to be conducted by practitioners or independent contractors, to facilitate data collection in order to measure outcomes that will feed into PD-MAP. The report provides a really impressive range of suggestions- linking strategic objectives to data – that ultimately reads like a set of obvious tasks finally compiled into one instructive document. By this point in the report, the PD-MAP comes across as an all-inclusive manual for how to conduct comprehensive PD evaluation; it’s an engineering solution to the big-tent of public diplomacy initiatives. This isn’t exactly a problem – but I’m not exactly sure how it helps other than putting it all in one place.
The report notes a number of possible measurements of effectiveness that can be researched after a particular intervention (like, say, a journalist training program) has occurred. It strikes me that what the authors suggest is probably already on the radar of those policy planners who do such programs, and who must justify their continued work. Put another way – shouldn’t practitioners engaged in particular interventions anticipate what would count as success? Or be in the best position to know? The authors rightly note the most logical signs of effectiveness – but how is this helpful other than telling program planners something they already probably know.
I don’t mean to be unfair to the report authors. I’m sure that many of the suggestions are helpful and perhaps, may not even have occurred to some practitioners. My guess is that the primary obstacles for practitioners in the field are twofold:
A) the ability to measure in ways that offer reliable (and unbiased) evidence.
B) the resources and time to do this kind of work along with all the other responsibilities placed upon people working in the field or in Washington.
And to be fair, the report authors also pretty much acknowledge this set of constraints. But the measurement suggestions, while often quite reasonable, are also at times arbitrary. And they rely on arguably shaky causation models. If, for example, we are concerned about US security and are interested in the effect of US PD efforts to minimize broadcasted statements encouraging violence against the US in a region that has been targeted by a PD intervention – how do we ultimately know that the PD campaign was the necessary, crucial, or sufficient cause. Simply put – we don’t. At least not based on this model.
The Value of PD-MAP
To be honest, I actually think this report is a comprehensive, systematic, if a bit idiosyncratic attempt to standardize performance metrics. In some sense, it reads like a catch-all field manual to direct attention to what “counts” as evidence – offering often reasonable, sometimes truly inspired, and yes, some logically dubious reductive attempts to demonstrate linkages between outputs and outcomes. It’s the Scout Handbook for Public Diplomacy evaluation.
So does improving US PD mean assigning values that can be standardized in order to compare across functional categories and to demonstrate particular objectives? My first read tells me that the value to PD-MAP is its deep tool-kit of data-collection ideas – not its elegance as model for analytical evaluation. Perhaps I’m reading it too harshly.
What I’m getting at here is that its most significant value isn’t knowledge creation, per se, but knowledge organization. The report touts the flexibility of the model, and the ability to handle various kinds of data, priorities, and risk factors that are very much context driven. When I first read this report, the first thought that came to mind was: hey, this is a great place to dump a bunch of data that I can play with and tweak as I see fit later. Sort of like a citation management software with lots of tag controls, or perhaps more accurately, a structured, relational database with some fun weighting functions built in that could allow me to code data effectively – like many of the computer assisted content analysis suites out there.
And then the obvious hit me. This is fine for a particular audience of policy planners whose job it is to provide reports about activity. It’s a project management tool. It’s for planners that need to look at the big(ger) picture. But the devils are in the details (and weights, for that matter).
My John Brown Moment
My day job as an academic puts me in a frame of mind to quibble with the method and methodological assumptions about insights derived from the data to be gathered in this model. I won’t bore you with specifics. My epistemological complaints aren’t that important at this stage when public diplomacy planners need a flexible, reliable methodology that would justify a set of methods workable for different applications. Evaluation divisions need marching orders to start the process of measurement. The PD-MAP starts to accomplish this, but leaves me with another thought. We need more disaggregated, separate investigations. I know this flies in the face of a lot of PD commentary on the need for real systematic data analysis.
John Brown provides the inspiration for this thought. Brown is a renown public diplomacy practitioner and commentator, whose incisive comments are a necessary ingredient in the discussion of US public diplomacy. Yet I’ve certainly disagreed with some of his arguments about the problems and futility of public diplomacy research. Brown seems to think that public diplomacy, like art, is a humanistic set of practices that transcends rigid disciplinary thinking and thus is diminished (perhaps dangerously so) by attempts to theorize about it. It’s politics, not political science. It’s a vocation, not a canonical academic discipline. That’s fine, but it doesn’t help those charged with coming up with something approaching at least nominally rigorous knowledge claims in policy-making contexts.
What I take from Brown’s position is not that we should discourage PD research. It’s that we should encourage interdisciplinary contribution to how we understand the very different processes that take place under the auspices of the term, public diplomacy. Brown is right to note that public diplomacy theorists (whoever they are) don’t have the “answers.” But scholarly methods more broadly considered can and do have a lot to contribute – especially when public diplomacy is really a combination of various different disciplinary forms of knowledge. What does this mean for PD-MAP?
Evaluation of US public diplomacy does not begin with PD-MAP. It may, however, end with PD-MAP – and that’s probably just fine. I think that those charged with PD evaluation and intelligence need to match the various situations, interventions (i.e. pd programs) and strategic objectives of public diplomacy with respective, established evaluative practices, theories, and experts. That probably means contracting work. But the DoD has done a good job of cultivating institutional knowledge centers in house… perhaps the time has come for State.
Public diplomacy is a big tent concept. In the United States, the term is an artifact of bureaucratic and legislative authority over the budget. As such, rather than reconcile different aspects under one evaluative umbrella, let’s treat them separately. If you’re going to measure media effects, commission media effects research. Don’t shoe-horn it into an a-theoretical database and call it a day. (To be fair, PD-MAP can be theoretically driven, assuming you tell it to act that way).
If the concern is, for example, influence outcomes – then cobble together experts and established measures of influence in order to flesh out a rigorously determined and tested framework. If something can be applicable across contexts or communication modalities, then ideally such frameworks can be used again. It’s not terrible if it can’t. It’s just inconvenient for policy planners who must point to particular precedence in order to justify future programs.
If the concern is understanding outcomes – then lets bring together the different kinds of expertise and ideas that reflect this concept outside of public diplomacy, that are already well-investigated in other disciplinary perspectives and professional contexts.
At a more micro level, if you want to know about the effectiveness of a particular media or information campaign – the PD-MAP is not the place to start. It may be the place to put your data once you’ve tweaked the model appropriately. But how, for example, do you infer outcome solutions when there may be complex analytical methods (like social network analysis) which provides certain kinds of evidence that you then must place in the context of other methods (like media framing or agenda setting analysis), in order to draw out implications for your strategic objectives? The imperatives of public diplomacy necessitate a mixed laboratory, so to speak, in order to really provide measurement and evaluation. It just seems like the PD-Map steps out ahead of the knowledge we need.
And I don’t mean to say that public diplomacy is so complex that we can’t ever really know if something is “working.” Nor am I saying that ideas and methods (or theories and methodologies) are not up to the task. I’m just saying that right now, the kind of knowledge required by measurement and evaluation units who must report to superiors is difficult to generate without a load of caveats. Which means the questions we ask of evaluation units may need to be less tightly bound to strategic imperatives, and more focused on very specific outcomes.
It may very well be that a truly deductive and predictive model for public diplomacy interventions is impossible given the range of circumstances, contexts, and priorities at stake. In this scenario, the most social-scientific approach probably will amount to a sophisticated structured case-comparison method, where multi-method cases of PD interventions are assembled and tagged in a useful way for policy-planners. Such an archive can allow policy-planners to reference a database of programs that yield useful clinical knowledge, matched to appropriate circumstances.
But the ambition of PD-MAP also suggests that particular, targeted quantitative analysis of specific public diplomacy interventions are not only possible but should be encouraged in order to amass comparative data. I agree. I just think we need to be more attuned to theory and already-established knowledge about persuasion, cross-cultural communication, media effects, and yes, sociological variables of collective action before we start inferring. I seriously commend the PD-MAP team for the ambition of linking outputs to outcomes. It is truly the holy grail of public diplomacy. We’ve got a rudimentary architecture, now lets fill in the house.
