Analysis: StateStat measures much, but delivers less than promised

By Charlie Hayward  

 

StateStat is a substantial part of Maryland state government’s transparency enhancements relating to performance measurement.

The program, an outgrowth of the CityStat initiative Martin O’Malley began as mayor, was much touted in the early days of the O’Malley-Brown administration, but now chugs along in obscurity.  While StateStat conveys much information, it can be substantially improved so that it comes close to living up to the high expectations inherent in the governor’s StateStat messaging.

O’Malley promotes StateStat as a tool for managing and reporting the state’s performance related to 16 wide-ranging, high-priority goals. Those goals are reported within four categories: Opportunity, Security, Sustainability, and Health. The Governor’s Delivery Unit (GDU) works closely with StateStat and Maryland agencies to monitor and track progress or deterioration for each goal.

O’Malley talked about the progress toward the goals at a long meeting with reporters in December.

Enhancing transparency

StateStat serves to enhance transparency in several ways. For example, it conveys to the public much of the same extensive data reported by the state’s departments and agencies to the delivery unit for managing performance, and it contains considerable meeting minutes and other materials documenting interaction among the governor, GDU, and agency managers.

StateStat’s website evidences large-scale and proactive work towards achieving goals by O’Malley, his GDU teammates, and state agencies.

StateStat’s goals seem reasonable, unbiased and ambitious. And there is good documentation of regular, active engagement between the governor and state agencies, including good information sharing, course modifications, and direction setting.  Moreover, the site is regularly updated to convey the latest executive direction and progress towards reaching goals. Indeed, as of March 2013 most goals’ delivery plans were completely updated, including more ambitious goals and new timelines.

Improvements could be made

However, there are many improvements the governor and his delivery unit team may want to consider.

1. Goals have long-term deadlines, from 2014 to 2025, so mismatches abound with the governor’s remaining 19 months in office, and between the extensive short-term activities reported in StateStat, vis-à-vis deadlines set far into the future.

2. High-level goals are supported by two lower levels of statistics and data. Users can drill down into secondary statistics and below them into detailed datasets. However, the audit trails are weak among the three tiers. Navigating from the 1st to 2nd and 2nd to 3rd tiers is not “user friendly” because the more detailed data mostly don’t agree with and don’t support data in the higher tier. Weak audit trails could confuse users and marginalize the value of the lower-level data.

3. StateStat’s Reports tab leads users to numerous meeting minutes, reporting templates, and graphs. This information is uncategorized; users who want to evaluate reports for a particular goal are out of luck unless they want to reconstruct from the content of reports and meetings they think may be relevant, forward to the relevant goals.

Some delivery plans missing; no audits done

4. StateStat has no delivery plans for  two of 16 of its goals Greenhouse Gases and Health IT.

5. State statutes governing StateStat provide that the Office of Legislative Audits should periodically audit the data. However, StateStat has never been audited and the most recent audit work on performance measures was July 2011. The auditor’s scope might include testing for perverse incentives that may be created by pressures on management to achieve numerical goals.

6. Based on a text-search of all laws passed in this year, there is only one reference to StateStat. This suggests StateStat is not being used by the legislature. That begs the question about who uses StateStat, whether the legislature is a stakeholder, and what is StateStat’s true value proposition compared to its cost. The governor may want to get legislators involved by soliciting their inputs on which goals to measure and what deadlines should be set.

7. One goal, Health IT, appears to be inactive since 2012.

Caveats needed

8. StateStat’s website should contain a section devoted to caveats about performance measures. First, many factors outside the governor’s control can have a direct and material effect on performance results. Second, performance measurement via statistics, while useful, is one of the least accurate ways of evaluating success or failure of government initiatives.

For instance, StateStat claims 132,800 jobs recovered since the great recession’s bottom (February 2010), or 91% of the goal. However, there is no way to know the true correlation between StateStat activity and jobs recovered and it’s conceivable the state’s other actions including last year’s tax hikes on the wealthy actually impaired job growth.

Performance auditing or program evaluations are expensive, but often are far more effective for evaluating how government is doing. Third, users should know that StateStat has no information about the cost-effectiveness of using taxpayer dollars for any goals being tracked.

9. The state, by law, reports extensive performance measurement data outside those available in StateStat. These “outside” data are reported annually in the state’s budget books. The StateStat website should link users to that additional measurement information and explain fully the relationships between the two measurements.

Conclusion

StateStat captures plenty of detail indicating the governor is focused on priorities embodied in his 16 goals. However, the site’s owners should recognize StateStat could be improved in many areas described above. Also, for those interested to see an alternative way to convey state performance, please compare StateStat against its counterpart in the State of Washington.

Charlie Hayward recently retired after 30 years’ experience with performance, IT, and financial auditing of a wide variety of government programs and activities. He is an avid reader of MarylandReporter.com. He can be reached at hungrypirana@verizon.net.