New results from impact studies

The ACRL Value of Academic Libraries site highlighted two articles recently published from the University of Minnesota, led by Shane Nackerud.  Both articles are published in the April 2013 issue of portal: Libraries and the Academy, although I'll be linking to their institutional repository.  The first article describes how the data was collected and analyzed and provides basic demographics of the users of library's services.  The research questions in this paper were:

• Do sufficient measures exist to determine what services individual library patrons use?

• Do the Libraries reach the majority of students in some way?

• Do students in different colleges use library materials and services in different ways?

• How does undergraduate library use compare to that of graduate students?

With increasing use of methods that capture, essentially, who uses each service, they were able to link with demographic and academic data about each user.  The "access points" or services for which data was captured includes:

• material circulation
• ILL requests
• library workstation logins 
• usage of electronic resources for those who were off-campus or those who logged into the library workstations
• attendance at workshops
• reference consultations
• course-integrated instruction (through Blackboard)

While not a complete set of data from all library services, this set does represent much of what the library provides.  Missing are usage of electronic resources from those on-campus and not using library workstations, in-house usage of materials, brief reference transactions, and visits to the library.  The authors admit that such data, especially the last, would be essential for measurement of "library as place", but they express concerns about over-reaching to such an extent that it would affect usage of the very services they would be measuring.  

But it's still a big data set - over 1.5 million transactions from over 61,000 unique users.  They were able to work with their Office of Institutional Research to get the demographic and academic data.  This step has been a common obstacle to such impact research for many libraries, whose own OIR's were reluctant to share the information.  The solution seems to be to split the collection of data between the two campus units - library gathers the user identifiers and the OIR provides the demographic/academic data, returning an anonymized data set to the library.  This way, neither party should have access to the entire data set, thus securing privacy that much more.  

With the complete, anonymized data set, the librarians were able to run correlation analysis to determine if academic achievement was in any way associated with library usage of any kind.  Pretty basic...and, to no one's surprise, there was a significant correlation.  This is very much in-line with other similar studies, such as the Library Impact Project.  I found this part interesting, though (emphasis added):

Already library staff have been able to share this data with University deans and administration and the feedback has been both positive and somewhat unexpected. For example, while University administrators have been enthused by the results, they are also not surprised. It seems intuitive that libraries should be able to demonstrate appropriate levels of usage, and that usage should result in increased academic success.

How frustrating!  University administration puts libraries (and others) under pressure to justify our value, our impact on student achievement, then says, "So?"!   Maybe this demonstrates a need to find out exactly what measures administrators expect.

In the second paper, the authors look at a subset of the population that could provide the clearest association of library usage and academic outcome: first-year, non-transfer students.  This group would have the fewest confounders, such as previous college experience, to muddie the results.  They looked at the effects of library usage on both student achievement (grades) and retention.  Their statistical analyses was more sophisticated than you typically see in LIS research, using not only t-tests and chi-squared-tests to determine the significance of differences between groups, but also determining the effect size (medium) and multiple linear and logistic regression.  They attribute the (relative) richness of this analysis to their own outreach to campus statisticians, and they recommend libraries not try to do it all themselves.  Hear, Hear.  The value of doing this analysis is that the authors demonstrated the size of the relationships they had found (significantly positive), but also the limitations of such relationships (most correlations were small).

However, with their models, they were able demonstrate significant effect of any library usage on GPA, while controlling for demographic factors.  Essentially, users of library service had a GPA 0.23 points higher than non-users, and that usage of the library accounted for 12.4% of the difference between the groups.  Given how many factors went into the model (12), that's a bigger chunk than expected (1/12th or 8.3%).  The second model broke the services out.  Not unexpectedly, these effects were much, much smaller.  The only services that showed statistically significant effects were database use, book loans, and workstation logins, and while the size was small, they were services that would be used repeatedly over the course of the semester. The totality of the usage of services represented a larger share of the effect - 13.7%.

Finally, their logistical regression models were conducted to predict student retention based on either usage of any service or usage of specific services.  This kind of model demonstrates the strength of the relationships by showing how library services can predict, or explain, outcomes.  This is a key aspect of research - can a variable predict a specific outcome?  If it can, then it can be used to change the outcome.  These models were both significant, even when adjusting for demographic factors.  Another important feature of logistic regression analysis is the calculation of the odds ratio (OR).  This measures the sized of the effect of the variable on the outcome.  In this case, students who used any of the library's services were 1.54 times more likely to continue to the next semester than those who did not.  Conversely, few of the individual services showed significant effects on retention; those that did were likely due to small sample sizes (few attendees).

So, what does this all mean?  Using these moderately-sophisticated statistical analyses is very much like triangulation - analyzing data from different angles to see the true picture.  This picture shows that there appears to be modest relationship between usage of any of the library's services and student achievement and retention.  However, picking out which services had the biggest effects is more difficult.  The linear model showed database logins, workstation logins and materials circulation as having a small effect; this doesn't show, though, in the logistic model.  More evidence, then, is needed.

It is somewhat disappointing that more interpersonal services, such as instruction and consultation, showed much lower effects.  This, I imagine, is due in no small part to the size of the data set.  Usage of these was services much lower compared to the more self-service, well, services. This could hide any association of the less-used services because of higher standard errors.  The problem with studying such low-usage services is selection bias.  If this can be controlled, randomly-selecting classes to provide instruction, then the effects should be more significant.

These articles, I think, are invaluable to the efforts of demonstrating value.  Like all applied research, it is but a piece in the overall puzzle.  It is not sufficient for the argument, but with more such studies filling in the gaps of knowledge, the picture becomes more and more clear.  It would be nice, however, if those who are the intended audience of such studies (presumably the campus decision-makers) would show their interest.