Revision - Transfer some current AE data elements to the Outlet file

[enielsen-air]

Name:
Evan Nielsen

State/Affiliation:
American Institutes for Research (AIR)

Description of Change:
Transfer some or all of the following data elements from the AE file to the Outlet file (or at least assess the feasibility of transferring):

• #501 Library Visits (VISITS)
• #650 Number of Internet Computers Used by General Public (GPTERMS)
• #651 Number of Uses (Sessions) of Public Internet Computers Per Year (PITUSR)
• #652 Wireless Sessions (WIFISESS)
• Also explore feasibility of transferring physical holdings and physical circulation

Justification:
With the existing PLS data schema, it is difficult to analyze library services and use at a local level within multi-outlet systems, which are predominantly in urban and suburban areas. Transferring existing PLS data elements from the AE to Outlet levels would address this problem and allow data users, especially those interested in urban and suburban areas, to have access to more location-specific data. These data element transfers would only affect multi-outlet systems, so the 80% of AEs in the PLS universe that are single-outlet would have no change to their reporting burden.

Potential Methodological Issues:
The current reporting method indicators for #501, #651, and #652 would also need to be moved to the outlet level. Furthermore, new imputation methods for the outlet file would be needed to fill in missing data, as most current methods only work with the LSA population value of the AE to create imputation strata.

States Already Collecting:
Georgia already collects all PLS data elements at the outlet level.

[mgolrick]

The burden on my libraries for this change would be astronomical.
There are (now) 68 AEs, but I have about 340 outlets. The data collection burden on each of the systems would be an increase in orders of magnitude. States where the number of AEs and outlets are close to the same would see no burden, states where the services are provided by many systems with multiple outlets would see a huge increase in burden. I am unalterably opposed to data collection at this level. My libraries would likely revolt.

[megdepriest]

California: 186 AEs, 1130 Outlets. Increases reporting burden significantly. 63% of CA libraries have more than one outlet. 27 AEs have more than 10 (664 outlets for those 27 AEs). That's a LOT of extra reporting.

[enielsen-air]

@mgolrick and @megdepriest , thank you for your feedback! I should have acknowledged that while 80% of all AEs are single-outlet, that proportion varies significantly from state to state.

That said, I have to push back a little: Am I wrong that these data elements are counted at the outlet level already? For example, how would one determine the number of visits for an entire AE without collecting the counts/estimates from each individual location and summing them. And @mgolrick , one of your catch phrases is, "I never let them do math." So why are visits or any of these other data elements an exception?

I think that even if we could just move the number of visits to the outlet level, that, along with HOURS and SQ_FEET could allow the ability to model the level of activity at individual library outlets, rather than only at the service area (jurisdiction) level.

[megdepriest]

I hear you, Evan! The burden lies more in the transition, I think. But manageable.

[rfuquaSLO]

This may be one of those rare instances where I find myself agreeing with @mgolrick 110%.

I would have a very difficult time being persuaded that the quality of data gathered through this potential change would be worth the increased burden. Also, I find this proposal a bit surprising given recent discussions about trying to better manage libraries' reporting burden in the context of the PLS.

I definitely get the general burden argument @enielsen-air has posed here -- 80% of Oregon libraries are single-outlet AEs, so NBD for them. However, the other 20% would likely show up at my door with torches and pitchforks if we tried to implement these changes.

Our rationale:
- #501 Library Visits (VISITS) - these are extremely messy figures. Last year (FY22) over a quarter of Oregon libraries either reported their counting method as an estimate, or reported that they do not track visits at all. Many of our locations and branches do not have automated gate counters, and those that do are constantly lamenting accuracy issues with them. It's messy data nationally, and I don't see outlet-level data improving that.
- #650 Number of Internet Computers Used by General Public (GPTERMS) - While I feel this is a critical thing for libraries to offer, I'm not sure what having outlet-level granularity of this data will help illustrate. It seems like there is pretty good evidence that libraries have reached "Peak-GPTERMS," given the pandemic, the cost of real estate, and the transition to a reality where more and more patrons bring their own devices.
- #651 Number of Uses (Sessions) of Public Internet Computers Per Year (PITUSR) - see above comment; a quarter of our libraries do not track this stat at all, or only provide estimates.
- #652 Wireless Sessions (WIFISESS) - again, VERY messy data here. Over 60% -- yes, you are ready that correctly, over 60% -- of Oregon libraries last year reported this number as either an estimate or did not track wi-fi sessions at all.
- Also explore feasibility of transferring physical holdings and physical circulation - Many of our multi-outlet libraries have floating collections (i.e., collections that are not necessarily assigned a physical location/branch). While I'm sure most of our multi-outlet libraries are tracking physical circulation by location, we would really need to hammer out some solid guidance for how to handle both item counts and circulation for floating collections.

Apologies for the diatribe here, but this proposal perfectly illustrates some growing concerns about the PLS we have in my state regarding data quality and scope. How much more detail and/or granularity can we continue to reasonably achieve through data that's mostly self-reported (mostly on a volunteer basis!) by over 9,000 AEs annually?

[enielsen-air]

Really appreciate your input, @rfuquaSLO. I want to reiterate that my proposal isn't definitively to move all of these data elements to the outlet file, it is just to explore which ones could be--so I listed the most likely candidates to start the exploration. As I said in my earlier reply, I think even having just visits moved to the outlet file would go a long way to understanding library use in a local context.

I really don't want to wear out my welcome in this wonderful community, but no one has yet explained why the burden would be so much higher to report the parts that they already have to collect to be able to report the sum that they are already reporting.

Furthermore, the fact that the visits figures are messy is a reason to capture them at the outlet level, in my opinion. If outlets are currently reporting estimates, they can continue to report estimates. But if AE figures are a combination of actual counts and estimates, wouldn't it be better to capture that than to have the sum as an estimate (which is the current guidance for the reporting methods questions)?

IMLS is currently exploring analytic methods that could enable measuring how libraries are related to strong communities. But those efforts are hindered by the lack of granular data for multi-outlet AEs.

[KristenCooke]

I wanted to chime in here. We are _very _granular with our survey.

Many of the libraries in our state may have only joined a consortium or system on paper while continuing to function very independently. Consortia in many cases in Arkansas are pro forma in order to qualify for state aid or formalize reciprocal borrowing. That creates a need for us to be more granular with our survey in order to contextualize the function and impact of our libraries better. For us, many of the questions that are asked of the AE on the PLS are also asked at the outlet level and auto-calculated to get a system total which is reported to IMLS.

I have a complicated relationship with our penchant for outlet-level questions mostly dependent upon what I am doing with the data at the moment. If I trying to understand a library's function within the community, it helps tremendously more than just AE-level data would. However, it does make the collection and submission to the portal a bit more complicated.

The most frustrating point is that any question asked at the outlet level and reported as -1 usually can require you to report the AE total as -1 as well. So, depending upon the edit checks, if one library in a system drops the ball, it could impact your ability to report a system total. If the problem is widespread, you can end up reporting less reliable data than when you were only reporting the AE total.

If we went this way, I think adding a reporting method question would be helpful (Estimate, Actual Total)

When I weigh the work it takes to ask granularly versus the benefit we get from it, I have to say I find it worth it. It has helped tremendously with capital campaigns, grant applications, and the like. However, we are a smaller state so I can't speak to scaling in larger states.

Kristen Cooke - Arkansas

[angelakfox]

Surprising no one, I would be extremely reluctant to see most of the elements considered here reported at the outlet level, largely for reasons that my colleagues have explained so well: increased burden on the larger systems; the possibility of making the overall data less reliable due to non-reporting or increased estimates; having to -1 entire systems for one missing branch, etc. But I think there's also something of an issue just with the optics of it all.

It speaks to Evan's point about how libraries must be collecting the data at the branch level in order to report the totals. Yes, libraries do need to collect that data somehow - but reporting them is something different. Reporting them is having that many more blank fields on the survey and needing to scroll down that much further to get to the bottom of the survey.

Reporting them is having the state tell you, “Yes, I know we've added so many questions in recent years. Yes, I know you’re extremely frustrated with all the new questions. Yes, I know you’re extremely frustrated with all the modifications to the new questions before you can even get used to them. Yes, we hear how burnt out you. Yes, I promise you we’re listening.
And yes, we’re still going to add even more questions – questions that seem to you to be completely unnecessary and without benefit.”

It feels somewhat tone deaf.

Our libraries are struggling. They’re continually asked to take on more work with fewer resources. Their libraries have become cultural/political battlegrounds and many find their boards are being weaponized against them. Adding a few more questions to the PLS (and telling them look, you're already collecting the numbers!) may seem like a small thing, but for some libraries, it's just an indication that the state isn't listening or doesn't care.

[enielsen-air]

Thank you for sharing that perspective, @angelakfox. The last few years have been a lot for everyone, but essential community service providers like public libraries have borne the brunt and, just when the pandemic recovery is in sight, the polarization conflicts ratchet up.

Yet the PLS historically has been a moving target; the most stable the survey has been in the last 15 years was the period right before the pandemic from FYs 2017-2019--only one change over the three years--during a veritable sea-change in the role of public libraries in their communities. So we're all just trying to make sure that the PLS can be as useful for the field in the next decade(s) as it has been for the last three and a half.

Since multiple commenters have now raised this issue, I feel the need to address the idea that collecting data at the outlet level would "decrease the reliability of the data" due to higher missing rates. My interpretation of this anxiety is that we'd rather maintain our plausible deniability of the current reliability problems in those data. Consider this: rather than an AE reporting a total visits figure that was missing one or more of their branch figures (and we don't know they are missing), if we instead collected visits at the outlet level and could impute the missing values, then the final dataset would likely reflect reality better than when we didn't know an entire branch's visits were left out of a reported AE total. I have to invoke @mgolrick again: "Don't let them do math!"

[KristenCooke]

@enielsen-air

Thanks for the comment about imputation. The crux of any concern/issue that arises for me was less the plausible deniability, but more indicative of my resignation toward some of the elements where I consoled myself that some data was better than no data. In the midst of my load-shedding, those seemed to be the only options. I remember now how the imputation process would resolve this area of concern.

[mgolrick]

I am concerned (among other things) about the burden. The initial increase in burden is on the individual libraries. And as someone whose average ratio of outlets to AEs is 5 and the range of outlets is from 1 to 22, there will be burden. Most of the burden will fall on the larger library systems who generally tend to have the capacity. However, having been on the other side of the fence, as a library director including multiple years in a system (AE) with 5 outlets, I can tell you that part of what would be required is a complete restructuring of the data gathering and reporting system.

The larger ongoing burden, however, is on the SDCs who have outlet to AE ratios which are large. This proposal has a huge increase on the amount of data with which we would be required to deal. Remember that each data element moved from the AE to outlet will, for me, mean that instead of 68 numbers to review, I will now have 340.

The larger issue that @enielsen-air raises is about whether we are counting the right things. He is not wrong. Are we counting the right things? I am not sure that dramatically increasing the level of granularity is as useful on a national scale. I would be supportive of adding, and actually for a few years collected, data on meeting room use. That is something that measures the value of the library as place in a community. We have not collected information on "self-directed activities" (aka "passive"). The pandemic showed me the fact that many people enjoy that aspect of the service that some provide. We are wrestling with counting the shift to digital. I think those changes are important and useful - especially as we look on a national, aggregated scale. I think that is part of the huge value of the PLS, the national-level aggregated data. Adding granularity simply makes our jobs harder.

We may not agree on some aspects of this, but I do think it is an important conversation to have.

[rfuquaSLO]

I'd just like to say a quick thanks to everyone for this discussion. I deeply appreciate (and want to echo) @angelakfox 's thoughtful comments about the realities our public libraries are facing right now, and want to reiterate that it doesn't feel right to keep asking more and more from them.
Also thanks to @mgolrick for acknowledging the increased burden on us SDCs that a change like this proposal could have on us!
Sorry if I am derailing the thread, @enielsen-air! This comment might reflect more of where I'm at personally with this work, but I wonder if we as a community are approaching a point of inflection with the PLS? Like, maybe there is a growing opportunity for us to have more discussions about the "big picture" of the PLS, what it is and where it should be going? I know a lot of us like to dive deep into the details with this stuff, but it might be a good exercise for us to stop and look at how we could improve the overall process for everyone involved (especially libraries).
The PLS represents a rich legacy of data we've all contributed to, and I'm not suggesting we chuck it all out the window.... It just sounds like a lot of us are all concerned about the sustainability of the effort, yet those concerns are often expressed in very different ways. Maybe a session topic for December? Who knows!

[enielsen-air]

@rfuquaSLO, you are not derailing the thread at all! It makes complete sense to me that this proposal has led to higher order questions about the scope of the PLS. I think Marisa would be open to starting a larger discussion in December, and the timing is good for her and the Mentors as they plan the SDC meeting agenda this fall. (Don't ask where it is, because I don't know!).

It seems to me that changing the reporting structure of metrics that are already tracked is a completely achievable objective, but it will require thorough understanding of the current systems used by libraries and SDCs to track and collect them.

If we want to be able to demonstrate, quantitatively, the role of libraries in their communities--not just as a way to access the latest John Grisham or Danielle Steel book for free, but as one of those essential "third places" where community members interact and learn about themselves, each other, and anything else under the sun--then we need to have more spatially granular data than we currently have for a county-wide (or multi-county) library system with 4-6 outlets. Maybe, as @mgolrick says, that is not feasible to do at the national level, but that just leads me back to @rfuquaSLO's question: what is the purpose of the PLS?

@mgolrick, I've been waiting for someone to propose the meeting room concept, as the LSWG discussed it briefly at the summer meeting this year. However, I have to ask: would you envision data elements about meeting rooms on the AE file or outlet file? IMHO, they clearly belong on the outlet file (at least a presence/count of rooms data element should go on the outlet file as a facilities metric).