A few days ago I was asked for some help with putting together a quiz that was on the Canvas LMS system. The person I was helping out was stuck for many hours on updating a quiz, something that you'd really badly hope would be easy in a LMS system.

Fast forward a bit and all hopes of an easy solution were lost. This proved to be an extremely frustrating experience because the preview window that was supposedly letting you see the version that the quiz takers would see had a cached version of the quiz that didn't reflect any of the recent changes that you had made. Not realizing that there would be problems with interacting with the system in multiple browser tabs/windows I opened one browser tab for the edits and another tab for the preview. I'd make changes in edit tab and they just never showed up in the preview tab. I found this very annoying so I looked into things a bit more and I found out that if you edit questions that are not in question groups the changes don't actually propagate to the existing quizzes.

This seemed non-intuitive to me but I can see why you might not want quizzes that may be a assessment task to be updated in the background. Except the behavior from question pools seems different. It seems like in the background one operation is very different to another but that the frontend is just a thin wrapper around an implementation detail I really don't care about.

Anyway enterprise systems often have little annoyances like this, but the big annoyance was yet to come. I try "fix" the question entries to match what the system wants me to do, which was suggested in a few posts I found online, and I go back to the preview and... nothing. At this point I wondered what was going on. I tried the other method of constructing the quiz questions and still nothing. So if you pressed "preview" all the state that was stored in the site was cached for that preview, if you subsequently made edits none of those would be reflected in the preview. The nasty part about this is that there's 2 cache related issues here at once and you can get one issue hidden by the other issue if you are unlucky.

It seems as though the design of this system assumed that there was only ever a single tab of navigation active at a single time. With that mental model the cache related problems may not have been so obvious to the developers. Having testers who would do things like check how the interaction with the system work from multiple browser tabs might have exposed some of these UX issues.

I'm not sure how exactly the preview works with this cache, if you press preview to create a tab T1 then make some changes, C1, then press preview to make another tab T2, then make some changes, C2, what do the preview tabs end up showing? Do they show C1 and C2 or both C2 or something else entirely? These are hard questions to answer with regards to caching, but I think the interface should either make it really clear how it's caching or just not cache at all.

If your system makes a lot of cached entries please make this obvious to your users

The way I was able to finally figure out how to accomplish what honestly seemed to be an easy task was to try to think about the system as it was implemented. The mental models that this gave me were what allowed me to make progress. But for most of the users of this system they have no mental model of caching or of how entities may have been stored in a database, for them the whole thing is utterly counterintuitive and frustrating. Despite figuring out how it worked I found it very counterintuitive, I suspect the only way I'd get used to this system is to make my workflow match exactly how the system implementation railroads me to do. I think this is a case of an especially poor design because the effects of the caching are not at all obvious.