Challenges on a Large Information Architecture Project (and a bit of levity for your day)

While attempting to write a case study, a colleague of mine asked me for some examples of issues I had on a recent intranet redesign project. This was a good project for examples, as it really was a fun and challenging project to work on. I thought I’d share the examples and hopefully (as a practitioner) you’ll find humorous, though the humour points to big problems!

The project was an intranet for call centre agents and some backoffice teams. The company was moving the intranet off of an old platform, onto a new platform, and at the same time adjusting the information architecture and rewriting the content. As I dove into the project with my research, strategy, information architecture, and taxonomy skills, it was obvious why they were redesigning the structure:

  • Thousands of pages, some out of date, all were long (and I mean if you printed them out they’d be about 10 pages of paper), organically grown (in the food world, the word “organic” has a positive connotation but not in the world of content!).
  • The page titles were very specific but the content on the page encompassed more than what the title indicated.
  • Content was separated by team, but teams needed the same content as each other. A specialized team might need a bit more detail, but instead of adding that detail elsewhere, the page was created again and content was duplicated, very technical, and hard to keep synchronized.

(And when you’re talking about duplicating a page that is 10 pages long when printed, then that’s some serious duplication. You might think that I’m exaggerating when I say 10 pages printed, so we could go ask my colleague, but if I had to underestimate, I’d say these pages were 7 pages printed.)

As an aside, a lot of times on projects like this, customers will say, “What we have is really bad. Part of me wonders if we should just scrap it all and start over. I don’t know how we’re going to restructure this.” I rarely think that scrapping everything is a good choice because I see existing content as a treasure-trove of material. In terms of taxonomy, the topics you’re talking about isn’t going to change whether you revise or start over. You can mine current content for a great array of things like topics, locations, audiences and all kinds of metadata like titles, descriptions, authors, dates. Even though the current content isn’t structured appropriately, it’s a basis and (dare I say) inspiration for a revised structure.

Back to the story at hand…

As part of the project, we planned a content audit, a taxonomy audit, card sorting, a new site map, and two rounds of task testing. To be totally honest, I only planned one round of task testing but the client said, “The better the structure is, the faster the agents can answer calls. We have access to the call agents and should do two rounds of task testing because it will lead directly to money saved.” I was somewhat shocked – I didn’t know what to say – because that was the FIRST time in 14 years of freelancing that a client had EVER said this to me. In response I said, “I agree. Two rounds then.”

I hummed along creating the site map, having a good time understanding this new subject matter and asking questions of the subject matter expert. We did card sorting with the agents and that was fun to see how fast or slow they sorted cards. We had the first version of the site map and did the first round of task testing. About half of our tasks succeeded and about half didn’t succeed. I was looking for a 70-75% success rate. When we looked at the tasks that didn’t succeed, we saw that for some questions agents interpreted the task differently than we anticipated and selected answers that made sense based on that interpretation. We adjusted our questions to be clearer for the second round.

For other questions, they simply failed: agents went to different area of the sitemap and selected different answers. Before the second round of task testing, we adjusted the placement of these pages within the site map. We ran the task testing again with this subset of questions and had about 6 questions instead of 12. About 5 questions succeeded so we knew the adjustments we made to both wording and structure worked.

However, one question still failed miserably!

The SME and I conjectured as to why the question failed, but I wanted to talk to the agents. We decided to talk to 5 call centre agents for 15 minutes each to ask about this question. I thought that this wouldn’t be enough time but I was working with what I had and kept my questions super focused.

Right about now is when I owe you some backstory: the question that failed was written so as to avoid a jargon term where new agents might not understand the meaning of this term and may not know to look up this term. We worded the question without jargon and we kept the title of the page as the jargon term. In the task testing, agents didn’t make the connection between the wording of our question and the title of the page, so I was concerned that the page title should be adjusted, too.

When we sat down with the agents and asked them the same question “If someone called in and said <non-jargon term> and wanted more information, where would you look for more information?” First, the agents said, “Do you mean <jargon term>? I would open their account and look at the <jargon term> notes.” I got a little chuckle at that. Here I was trying to avoid jargon, but both new and experienced agents understood this jargon term. But you don’t know unless you test!

So what do you do when you have 15 minutes with a user and you’ve used 2? Do you let them leave the room? No! You keep asking questions. I explained in more detail what I was working on and why. Given that the intranet was their lifeline for answering customer questions as quickly as possible, they were understandably concerned about changes to this tool.

All five people I interviewed said, “Whatever you do, please make sure that CTRL+F is still available.” (Inside I did another little chuckle because CTRL+F is a browser feature, not an intranet feature so I knew that any redesign would have CTRL+F.)

I couldn’t not dig into that comment! I asked “Why is CTRL+F important to you?” They explained that they used it in a couple scenarios:

  • One was that they would use the search box on the intranet to search for a term and the search results would show 50 items but the term wasn’t necessarily in the title or the description displayed in the results.  If they clicked on the search result and got to a content page, they would then use CTRL+F to quickly search for that term on the page.
  • The other scenario would be that if they browsed to the page or accessed the page from a bookmark. Once on the page, the used CTRL+F to quickly find the term.

To me, this behaviour was fascinating because it showed several things:

  • New and experienced agents all used this same search function, so it must have been a skill taught to all agents.
  • The search results were not showing relevant information.
  • The page titles and descriptions were not useful.
  • The pages were waaaaaayyyy toooooo looooonnnggg.

Breaking apart these pages, as we planned in the new IA, would fix almost all of these issues: page titles and descriptions would reflect the contents of the page, the search results could show these relevant results, and once on the page the agent would be able to quickly see the relevant content. And they could still use CTRL+F.

Sometimes it’s amazing what you can do with users in 15 minutes and when you dig in to rather innocuous comments…

The last interesting story on this project (or at least the last for this article) was that the subject matter was so vast that and deep that different teams had different definitions for the same terms.

For example, customers made payments and had questions about payments, so call centre agents used the payments term in relation to customer payments. The Payments team investigated payment problems and resolved back office issues. When I asked the Payments team to card sort payments topics in an in-person card sorting, it turned out that they didn’t recognize many of the payments topics and that their payment topics were mostly related to general ledger reconciliation and back office adjustments. For them, customers were not involved and any customer related payment content fell into the Billing category. (Which I found fascinating because while one can receive a bill, one makes a payment on that bill and if something happens with the payment, then someone is going to ask about the payment, not about the bill.)

Because the intranet was targeted at the call centre agents that needed quick answers while on the phone, we decided to broaden the term payments to include more than the payment team-specific content. Given the small amount of content that the Payments team had and that it was for occasional reference, we decided to re-appropriate the term back into a general term. There would be content that only the Payments team needed and that the agents shouldn’t see, but we decided to address that issue by restricting content based on permissions.

The moral of the story: no matter what happens, keep digging into the why. Never assume that you know why things are happening. Even if you think you know, ask someone, “Would you mind explaining that to me?” Or “I’m interested in why you say that. Can you tell me more?” Curiosity is key!

About the Author: