|
|
|
Week 9: Measurement Universities measure an almost endless variety of things associated with their operation. They count students, credit hours, faculty, and staff. They produce data on the budget, expenditures, the number of graduates, the number of admissions, and the number of applications. Universities can tell you graduation rates, persistence rates, part-time vs. full-time students, number of fraternities and sororities, amount of sponsored research, grants received, grants applied for, and doctorates granted. Universities will provide data on athletic attendance figures and won-loss records, number of students employed after five years, starting salaries of graduates from the business school, and the number of alumni making donations. They know how many credit hours it takes to get each degree, the number of classrooms and their capacities, how many of their course enrollments are small or medium or large, and how many square feet of the campus represent recreation area. In short, universities collect an incredible amount of information. Furthermore, universities provide reports to an extensive list of external agencies, although public universities report more data than private universities. These reports satisfy questions about crime on campus, ethnic and gender diversity, athletic participation, and graduation rates of student athletes and non-student athletes. They respond to concerns about human subjects and animal rights, fraternities and sororities, scholarships or fellowships or other forms of financial aid, and overhead costs for research. In every state, public universities respond to an mind-numbing list of special data requests to answer specific bureaucratic or political questions generated by state agencies, governors' offices, and legislators. Universities provide data to accreditation agencies, ranking organizations, disciplinary associations, and university lobbying groups in support of one or another agenda. The volume of this data is truly amazing. While most universities have institutional research offices that manage significant parts of the data flood, many other offices of the university participate. The accountants generate accounting data, professional schools generate specific data for their specialized accreditation agencies, athletic programs develop data sets for their conferences and the NCAA, the registrar produces student data, and the alumni association and fund raising offices develop focused statistics on their constituencies. In most universities, a rigorous examination of all the data will produce despair. Each office produces data for the specific purpose intended and often uses slightly different definitions, calculations, or data universes to deliver the results. If we compare, to take one example, the data reported to various agencies on annual giving and endowment and the same figures appearing in the university's official documents, we will get significantly different numbers. None of them is wrong, but all of them understand the question differently. Do we mean cash in hand, do we include deferred gifts, do we use the present value of these gifts or the face value, do we define the premium paid to purchase sports tickets as a gift, do we report matching dollars from foundations or the state, do we include the unrealized gain on the endowment, do we deposit cash operating surplus into endowment, and do we report the endowment as of June 30th or January 1st? All of these variations and many others illustrate the peculiarities of university data. For student data, in complex universities, uniformity is also difficult to achieve. Ask a registrar for the university's enrollment. Not an easy answer. Do we want fall or spring or summer or an average? Do we want fall enrollment on the first day or the 14th day, after drop and add, in the fifth week, by the fee payment deadline, or at the end of the semester? Do we want full-time or part-time enrollment or headcount enrollment? Do we define full time as 15 credit hours per semester or 12 credit hours per semester? Each university or college has its conventions for reporting these things and the different agencies requesting the data have their requirements as well. As one might expect, the conventions differ from university to university and from agency to agency, rendering many comparisons based on these data difficult to interpret. Even when we can agree on a definition, comparative interpretation may also prove difficult. Graduation rates are a favorite data point. Intuitively people like this number as it seems to tell them something about the success of the university in educating undergraduate students. In fact, however, graduation rates have almost no validity as a comparative measure for colleges or universities with any differences in student characteristics. If one university has 50% of their students studying part-time and another has 90% enrolled full-time, we can make few relevant comparisons of graduation rates since part- and full-time students have different paths to graduation. Student characteristics make a large difference in graduation rates. Wealthy full-time students who pay a high tuition or have a time-limited scholarship have high graduation rates compared to students in the same institutions who do not share these characteristics. Often, in today's world, public institutions have many students who transfer to a different institution closer to home or with more desirable programs. These students, even if in good academic standing when they leave, are counted as failures for graduation rates. Today's universities also have many students who transfer into the institution from community colleges or other institutions. None of these transfers when they graduate are counted as successes in the graduation rate which only uses the first-time fall enrolled students as the basis for the calculation. As a result, when observers cite comparative graduation rates to make a point about college performance, well-informed people discount the argument. Similarly, anything that divides by the number of faculty is likely to produce a spurious comparative measure. Student-faculty ratios, for example, help institutions promote their campuses to the public. Most large research universities have low numbers here: 11 students to each faculty member perhaps. So too do most small liberal arts colleges. What does this number mean? Nothing, because the context of a liberal arts college and a large research university are much different. Even among large research universities, the actual interaction between faculty and students is much less a function of some global statistic like a faculty-student ratio than it is of curricular organization, teaching mission, and the definition of what constitutes a faculty member. If faculty members include librarians, extension agents, and clinical faculty from the medical school, few of these faculty will have any impact at all on an undergraduate's opportunity to learn directly from a faculty member. Productivity measures that divide credit hours, research expenditures, or any other output quantity of the university by the number of faculty members is almost sure to deliver misleading results. One university may have 4,000 faculty and another 3,000 faculty and produce the same amount of research funding. Is the larger university faculty less productive than the smaller one? Not necessarily, since the larger number of faculty may include extension agents, librarians, and clinical faculty who have no research assignment at all while the smaller faculty may all be academic faculty with a research assignment. Finally, universities, especially major public and private research universities, differ significantly in the structure of their curriculum and the number of disciplines they support. The existence of a medical school, a hospital, a land-grant agricultural mission; an emphasis on arts and humanities or physical and natural sciences; the presence of an engineering college or a major business school--all these will have an impact on comparative data that often renders them impossible to interpret when comparing institutional performance. None of this exempts the university from measuring what it does, quite the contrary. Universities must measure what they do if they want to improve, but they must collect and use data carefully to avoid misleading themselves. University data actually have at least two, quite distinct audiences.
Universities often confuse these two different data universes. They will take the data developed to defend the university against a legislative assault on its effectiveness and think that it represents a metric on which the university should drive its budgets. Often this is inappropriate, not because the data are themselves inaccurate (although they may be), but rather because the data express a reality in terms defined by the political context of a response to a legislative attack not in terms related to the institution's need for improvement. For example, if the legislature says that the faculty do not teach enough, a common complaint, the university will generate a host of information on the role of the faculty in the classroom. This is fine, this may deflect the attack, but data on the number of hours the faculty spend in the classroom, on the number of students who have interactions with faculty in groups of fewer than 50, or similar statistics, often do not help the university actually improve the curriculum, enhance the student experience, or effectively use the faculty resources available to the institution. Universities will often seek out rankings that demonstrate how well they do, even when they know the rankings data to be suspect, the methodology unsound, and the results unreliable. Nonetheless, because their constituencies have an insatiable hunger for evidence of institutional distinction, universities will promote and distribute rankings of dubious value simply to meet that need. While there is a certain naive charm to this form of self-promotion, it becomes a problem if the institution comes to believe its own propaganda and assumes it does well because its public relations office says so. It may be doing well, but not because the suspect ranking or television advertising says so. What then can a university to do about measurement? The university can begin by recognizing that in developing comparative institutional indicators, a few measurements generally are better than many. This is because a few measurements will serve this purpose well, and little additional improvement in accuracy comes from adding additional, more complicated measures. For research universities, the total annual institutional expenditure on research from federal sources is by far the most reliable measure, especially when augmented by the total annual institutional expenditure on research from all external sources. These two give a scale for the university's research enterprise that, while not resolving all the possible data problems, gives a good indication of institutional strength among major research universities. The Top American Research Universities publication provides a good example of this approach. If the university wants to improve its performance, it needs to measure itself in a consistent fashion, using a few key indicators and then track improvement. There is no substitute for this process, for unless the university has a reliable and consistent way of measuring its own performance, it cannot measure improvement. If it cannot measure improvement, it cannot reward it. Absent rewards based on clear and consistent measurement, improvement will happen in an idiosyncratic fashion if at all. University internal measurement falls into two general categories: The global indicators of performance, and the operational measures of effectiveness. These are not the same type of measurement.
These distinctions are similar to business data that provide the firm's board and investors with a few global indicators of profitability such as stock price, price to earnings ratio, and similar information. Operational indicators, however, help managers track sales targets, cost per unit of production, turnover time of inventory, cost of credit, efficiency of production, and similar process variables that contribute to the global measures of firm success. Boards and investors may look at a only a few global comparative indexes to track the firm's success. Not only do they want the stock price to rise, but they want it to rise faster than the price of similar firms in the market. They want to know the market share of their company, they want to know the performance of their company relative to others in the same business. Again, exactly how the firm succeeds in delivering good results is an operational issue for internal management. What matters to the success of the firm are the results visible through improvement in the global indicators. It is management's job to get the results, and managers in different circumstances will use different techniques to achieve good results. What counts, however, are the results. For universities, the results measures are the global indicators of performance, and the operational measures are what managers use to ensure that they can deliver the results. Value budgeting, described earlier, implements these principles. A value budget might well begin by dividing the university's products into two categories: teaching and research. It defines the responsible units for this teaching and research as the schools and colleges of the university. It makes no assumptions about the responsible units beneath the level of the school or college, leaving this operational domain of departments and programs to the management of the responsible unit administration. Having defined these two products--teaching and research--the university then creates a method for independently measuring the productivity and the quality of each product. Through its value budgeting system, the university also develops a cost allocation system that assigns all direct costs of the colleges to either teaching or research. This is an important simplification that serves to highlight institutional values. It asks the question: "what is most important for this university?" The answer is "teaching and research," although universities may chose different values to define as most important. In this example, we then must ask how to account for the costs of academic advising, community service, and the host of other things that universities do. Value budgeting answers this question through the following analysis. Value budgeting recognizes that the university's values actually exist in a hierarchy.
By putting all costs into either research or teaching, the university gains a clear means of evaluating what it does and makes explicit its values. If, for example, an institution has an opportunity to participate in a community service project that will take three faculty members two weeks of work, value budgeting recognizes that this effort represents a cost to either teaching or research or both. The community service may well be a good thing to do, but it will cost the university to do it. By allocating the costs of projects, programs, and activities to teaching or research, the system forces everyone to make clear choices and also forces everyone to recognize that faculty and staff work and time are valuable and limited goods. If the university assigns faculty to do two things, teaching and research for example, then when it chooses to reassign the faculty members to work on other projects it must, unless the faculty member chooses to work for free, reduce the faculty member's commitment to the first two assignments. This makes the cost of teaching and research higher because no longer is 100% of the faculty's effort devoted to teaching and research but instead is reallocated to other activities, perhaps community service or departmental administration. Again, this is not a bad thing, it is just not a free thing. Many academic people and their constituents dislike this conversation because it seems to deny the social mission of the university. This system actually supports the social mission of the university because it requires both university and society to be clear about what they want. Do they want the faculty working full time on teaching, on research, on public service, on administration, or on some combination? The problem arises when the university, in response to the impassioned call for public service or the desire to have more detailed administrative controls, diverts faculty effort to these purposes and then discovers that others in its constituency think the faculty do not teach enough. When the institution's values and measures are unclear, universities find themselves attempting to meet everyone's demands without regard to resources. The result, naturally, is that more is expected of the university than it can possibly deliver, leading to an undeserved sense that the university is not an effective organization. Actually, it is effective but unclear about the use of its limited resources. Universities that seek to please by accepting obligations without the mechanism to make explicit the cost of these new obligations can be popular in the short term but will eventually lose the ability to be competitive because they spread their resources and dilute their effectiveness. To implement a value budget, the university must measure teaching and research. Teaching has many forms and structures in a university, and measuring the productivity and quality of teaching poses some interesting challenges. Academics can imagine resolving some of these by designing unique and specifically tailored data systems to capture every nuance of the teaching process. Normally, while such projects offer interesting and important intellectual challenges, they do not provide practical alternatives usable in real time. Instead, universities usually find they can get close enough to the issue of productivity using whatever accounting mechanism already exists for tracking course work for other purposes such as assigning work loads and certifying completion of degree requirements. Credit hours in most colleges and universities represent the coin of the realm for instruction. Students define their progress and the faculty define their work load in credit hours; students and parents pay for credit hours; states pay for credit hours; and the faculty and accreditation agencies approve curricula in terms of credit hours. For these reasons, most universities have good credit hour data, and they can generate almost any information needed about credit hours. They can show how many credit hours students take on the way to graduation, they can calculate how many each faculty member teaches or how many credit hours each department produces. The value budget takes as its teaching productivity indicator credit hours and assigns credit hours to the college that pays for the faculty time required to deliver those credit hours. The key here is to assign the productivity to the same unit that bears the cost of generating that productivity. If a history professor teaches an education class, the credit hours belong to history unless the college of education chooses to pay the history department for the work of the professor. A value budget also must address the question of differential productivity between large and small classes, and perhaps the differential cost of high investment classes in the sciences and lower investment classes in the humanities. In practice, however, while very elaborate weighting systems give comfort to those who think fine distinctions important, they actually make little difference in the results when dealing with complex colleges. At the department level it is possible that differential weightings by type and intensively of class might make a difference, in the scale used for value budgeting, the only productivity weighting that appears helpful is done by class size. This weighting, like all such weightings, reflects a judgment, an academic value. The academic value is that the university should not provide incentives for large or small classes but should structure the system so that decisions on class size primarily reflect pedagogical issues rather than fiscal issues. To do that, the value model assigns a higher weight to small classes than to large ones. The size of the weight is pragmatic, determined by an analysis of what weighting would produce the closest balance between large and small classes in the measurement of improvement. The goal is to make small classes worth doing, but not enough that a unit could succeed in improving its productivity measure dramatically by reallocating all its faculty effort to small classes, a strategy that would produce low overall productivity. Similarly, the goal is to make large classes worth doing, but not enough that a strategy of assigning most faculty effort to large classes would be more advantageous than a strategy that mixes large and small classes. In practice, when installing a value budgeting system, some participating units will attempt to game the system. If the value budget is carefully designed using good current data, the weights will be such that the university can demonstrate that an over emphasis on one or another class size will not produce optimal productivity or give an advantage in earning the associated rewards for improvement. Research is the other top level activity of the university. The measure of research productivity is much more complex than the measure of teaching productivity because no easy data element captures most faculty research in the same way credit hours capture most of the university's teaching. Nonetheless, for the university overall or for college- or school-level measurements within an institution, the total expenditures on sponsored research serves as a good surrogate. These data are available and unit managers understand them well. Some units will have much larger opportunities and achievements in research funding than others, but the value budget's evaluation process resolves this problem as described below. Sponsored research expenditures serve quite effectively for the purpose of identifying research productivity. The value budget also tracks fund raising and other income at the top level of productivity evaluation. Both indicators measure success in acquiring funds other than those generated for sponsored research or teaching (principally state and tuition revenue) in support of the university's missions. Fund raising refers to the development/advancement activities that solicit private gifts from individuals, corporations, or foundations on behalf of the university's missions. Other income addresses sources such as clinical revenue, patent and license income, and some other forms of service revenue such as the income from the sales of agricultural products or from hosting conferences. As is the case with all explicit measures in value budgeting, these reflect institutional priorities and makes them visible. The university recognizes that its institutional opportunities for improvement depend heavily on the expansion of income sources beyond those from research funding and state/tuition support. For that reason, and to generate the incentives necessary for this effort, the institution includes fund raising improvement and other income improvement among its top indicators. In value budgeting, a common concern has to do with scale and opportunity. When universities establish any standardized measures, the various guilds and colleges immediately attack them as inappropriate or unfair for their particular guild. If the productivity number is credit hours, small units teaching foreign language courses in small classes will feel disadvantaged in comparison to business schools teaching many students through technology enabled education methods. In systems that compare the total productivity of colleges and other units and reward those that have the largest share, these complaints have value. One of the great strengths of value budgeting is its solution to this problem. The university makes the academic judgment that each college or major unit is valuable for its own contribution to the institution's programs. If the university chooses to eliminate a unit or establish a new program, the value budget is not the mechanism to accomplish such a major redefinition of mission. The value budget deals with improvement in the university's existing programs, not with the academic rational for program invention or termination. Within that context, then, the university uses the value budget to create incentives for individual college improvement until each college reaches the highest possible level of performance. At that point, value budgeting rewards the maintenance of top level performance. Productivity data serve to measure each college's improvement against its own previous performance. Value budgeting does not care whether architecture has as many credit hours as business. What matters is whether architecture has improved its own performance against its prior year's measures, and whether business has improved measured against the prior year. Every unit CAN get better and receive a reward. Not every unit does of course. Given this value, the data serve to measure each college or school's improvement against its previous performance. If a college gets better, it deserves a reward. The same principles apply to research productivity. A research intensive college of medicine should have high research productivity because such colleges are major players in the federally-funded life sciences research marketplace. Colleges of fine arts, however, will have small research productivity measured in grants because the market for this activity in the fine arts is small. Nonetheless, if Fine Arts improves from a $500,000 externally funded grant portfolio to $1,000,000 by virtue of successful application for grants and foundation funding, it will be more deserving of a reward than the college of medicine that remains stagnant at $20 million of research funding when comparable colleges around the country may well earn twice that amount and grow each year. They key for all of value budgeting is to improve within the national context of the college, not in comparison to quite different colleges. In the aggregate, if each of the colleges or units within the university improves within their context, the university at large will also improve since the aggregate performance of the university comes from the locally driven performance of the schools or colleges. It is this college-based performance that value budgeting must reward. This part of the conversation about measurement focuses on productivity, for productivity is half of the equation for value budgeting. Productivity measurement ensures that the colleges and the university get the most from the money available. Money, as all university people know, is required for quality and effectiveness. Money buys the people, the support, and other materials needed to do academic work. The more money the university has, the more it can do. Productivity improvement is equivalent to new money. Productivity improvement means doing the same work for less money or more work for the same money. Every dollar generated by increased productivity offers an opportunity to invest in quality, initiate new programs, or support faculty and student projects. Among the various sources of new money to enhance university functions, productivity increase offers one of the few that is almost completely under the control of the institution and its academic and administrative units. However, productivity alone does not suffice for driving a value-based budget. Quality is equally important. Productivity without quality produces a poor result. If the incentive is for productivity alone, universities will drive quality down to the lowest acceptable level because no reward attaches to improving quality and all rewards go for productivity. For this reason, a value budget rewards improvement in quality at the same level as improvement in productivity. Once a unit meets its base line performance criteria, improvement in quality or productivity produce separate rewards; improvement in both produces a double reward. Often, it turns out, the college finds that by improving productivity, it also improves quality because by being more efficient and effective it also does its work better for students and for research. As colleges improve, value budgeting provides dollar rewards the college can use to reinvest in additional quality for their units. With this background and the readings, we can engage the debate on these and related issues:
© 2018 |