Column: Survey Says, “Help Us Design One”

The Chronicle was among 3,000 Ann Arbor households that have received three pieces of mail over the last couple of weeks sent on behalf of the city of Ann Arbor by the National Research Center in Boulder, Colorado. First to arrive was a post card alerting us to the fact that our household had been “selected at random to participate in an anonymous citizen survey about the city of Ann Arbor,” and that we should watch the following week’s mail for the survey and instructions.

As indicated in the postcard, the materials arrived the following week. With the survey itself in hand, it was possible to determine the context of the survey in a bit more detail. It turns out that this is the same survey that was conducted last year (2007), and is the basis of the Voice of the People awards to the city’s recreational services and to the Ann Arbor District Library from the International City/County Management Association, which were presented earlier this year.

A call to the National Research Center and a conversation with the manager in charge of Ann Arbor’s project, Damema Mann, confirmed some of the nuts and bolts of the survey protocol:

• The mailing was in three waves, consisting of a post card, a first survey (blue), and a reminder survey (yellow)
• The survey forms are not optically scan-able, which means that results are tallied via data-entry keying by hand.
• Responses to the open-ended question, “Please finish this sentence: If I could change one thing about the city of Ann Arbor, it would be … ” are recorded complete with spelling and grammatical mistakes. Profanity has not come up with Ann Arbor’s survey, as best as Mann recalls, but for their public reports, the city would have the option of asking that symbol characters be substituted for profane words.
• It’s not possible to filter out responses from households that fill out both the initial survey and the follow-up reminder because the forms aren’t coded with the address to which they’re sent (which preserves anonymity of respondents on the survey vendor’s end). That also means there’s no “pull” between the first and second survey mailing, to prevent the second mailing from going to people who’ve already responded.
• It’s not possible for the city to prevent mailing of surveys to specific addresses where known malcontents live, because the only data provided by the city is a set of zipcodes. The addresses are sampled by the survey vendor from GIS data generated from these zipcode ranges.
• The survey is a template used nationally (which facilitates comparison with other municipalities) with a custom question at the end. Ann Arbor’s custom question asks respondents (i) to rank a range of investments like open space, public safety and online services on a scale of importance from “essential” to “not at all important”; (ii) to indicate what resources they use for news about the city of Ann Arbor; and (iii) to rank a range of factors as related to their quality of life in Ann Arbor, like bike paths, strict code enforcement, and preservation of historic districts, on a scale of importance from “essential” to “not at all important.”
• There’s no internet option for Ann Arbor’s survey. It’s not recommended as the sole response alternative (not everyone has access to a computer) and when it’s added as an option, it costs extra.

If it costs extra for an internet survey, how much does the the city of Ann Arbor pay for the administration of this paper survey and compilations of its results? According to information provided by Lisa Wondrash, who is communications unit manager for the city of Ann Arbor, the basic service for a survey that is mailed to 1,200 residents is $9,600. But the city of Ann Arbor expanded the distribution from 1,200 residents to 3,000 and added the open-ended question above. The additions cost $8,100, bringing the cost of the survey to $17,700. The cost includes benchmarking Ann Arbor against other similar-sized communities across the U.S.

Teeing Off: An Editorial Aside

I’d like to encourage Chronicle readers who have received the survey described in this article to complete and return it. Last year the response rate was around 38%, which is towards the high end of the range of response rates nationwide. You might as well do it – we already paid for it.

But in the context of a recent memo sent by city administrator Roger Fraser to city staff asking them to think about ways to achieve cost reductions in their departments of 15% over the next two years, I’d like to suggest that this survey is a good candidate for an expense to be cut. Yes, it’s important to solicit from citizens their feedback that might be critical, and anonymous surveys are one way to ensure that citizens can submit their feedback without fear of any kind of reprisal.

But we also have something called democracy, which provides that citizens can participate in decision-making by freely expressing their opinions. Perhaps that’s naiveté on my part to think that it should not be necessary to afford anonymity to citizens in order for them to feel comfortable expressing critical views in Ann Arbor.

I think that for the city, the perceived benefit has less to do with the information gained from citizens per se than it does with the fact that part of the set of deliverables is the possibility for awards – awards of the sort that the city’s recreational services and the library received from the International City/County Management Association last year. In general, I would submit that there is undue weight given to awards, certificates and rankings in Ann Arbor’s public discourse. The currency we trade with is too often awards and certificates. One good example is mayor of Ann Arbor John Hieftje’s response to a question during the fall mayoral debates aired on CTN about what leadership skills he would bring to the office of mayor. He cited awards he’d won for leadership.

Part of the reason the blog (when still actively maintained) Ann Arbor is Overrated resonated with its readership was not just the sense that Ann Arbor was rated too highly in some ranking or other, but the sense that Ann Arbor is over-ly rated, as in too often – ratings and rankings cashed in for rhetorical points, as a substitute for a realistic assessment of how much we have left that we can and must achieve. A digital cities award does not assuage my frustration when I cannot easily find the email address of a staff person at the city when I’m searching on the city’s website and I know the name of the person.

Here at The Chronicle we’ve published a couple of New Media Watch items in our first three months citing announcements of Ann Arbor rankings in other publications. So to that extent at least, we’ve been complicit in the acceptance of awards and rankings as a legitimate currency of conversation. Perhaps we should contemplate a news blackout on rankings and ratings.

At the Dec. 8 city council working session, administrator Roger Fraser praised the efforts of city staff in their apparent initial success in changing the financial direction of Leslie Golf Course. The measure of success was not an award or a certificate, but rather the bottom line. One could argue that the real key was not the hard work of staff but rather the liquor license that city council had awarded to the course. But I think Fraser got something right in connecting the staff’s energy and renewed sense of commitment to the opportunity council had provided to make the improvements necessary for successful golf at Leslie. What he didn’t do was connect the renewed enthusiasm he’d observed among staff to an award or certificate they’d received.

Looking past this year to the next two years, I don’t think the potential for awards gives us $17,700 worth of value. But if Ann Arbor doesn’t participate in the survey, surely some amount of useful information might go missing.

In an attempt to make up partly for that information deficit, I’ll suggest that we put together an online survey available here on The Chronicle to serve two specific purposes. The first goal would be to add citizen input to Fraser’s directive to staff: how do you think we could save 15% over the next two years? The second goal would be to supplement the picture of Ann Arbor that the National Citizens Survey gives.

So until the end of the year, we’ll take suggestions in the comments section of this article for survey questions along those lines. We’ll launch the survey on Jan. 1, 2009. Thanks in advance for your help.