{"id":2304,"date":"2019-02-28T10:25:26","date_gmt":"2019-02-28T09:25:26","guid":{"rendered":"http:\/\/www.stevebromley.com\/blog\/?p=2304"},"modified":"2019-11-06T17:44:57","modified_gmt":"2019-11-06T16:44:57","slug":"some-things-ive-learned-as-an-in-house-research-lead","status":"publish","type":"post","link":"https:\/\/www.stevebromley.com\/blog\/2019\/02\/28\/some-things-ive-learned-as-an-in-house-research-lead\/","title":{"rendered":"Some things I\u2019ve learned as an in-house research lead"},"content":{"rendered":"

For the last 2.5 years, I have been user research lead within a public sector organisation (as evident from the blog posts\u2026<\/a>). This was my first time being a line manager and my first time introducing research into an organisation that didn\u2019t do it previously. During both of these experiences I made some mistakes, and feel like I learned a lot. Most of the lessons (or thoughts cribbed from colleagues smarter than me) are probably only relevant to me, but I thought I would document them in case anything I learned is of value to others going into similar contexts.<\/p>\n

I\u2019m also conscious that this will reflect my own experiences, and is going to be subjective \u2013 perhaps there won\u2019t be many universal truths in my own reflections, and I\u2019d recommend reading other sources before drawing conclusions. In no particular order, here\u2019s some thoughts\u2026<\/p>\n

One thing I learned: Evidence only works when you have agreement on what you\u2019re trying to achieve<\/strong><\/p>\n

For a long time after I joined, we were following a development strategy that had one principal goal \u2013 to produce a HTML representation of information generated by the organisation, mirroring the structure of the concepts it represents, and enable others to build new things using this output. I think this is called the semantic web. This approach has a lot of benefits \u2013 for example the points made at the end of this blog post<\/a> about helping SEO, and enabling others to use your data to build smarter things than you can imagine yourself. However this approach was different to the one being advocated for within the product teams, of understanding who uses our existing website and designing useful tools and services to meet their needs. Both approaches can be described as user centred \u2013 they just didn\u2019t agree on who they wanted the users for a new website to be, and ultimately what \u2018the website\u2019 should be.<\/p>\n

The impact of this lack of agreement was that a lot of the research we ran early on felt like it was having low impact. Our work was guided by product teams and we would generate a lot of evidence that the pages weren\u2019t meeting the product team\u2019s goals \u2013 but because the development approach had different goals than \u201cutility for existing users of the website\u201d, that evidence wasn\u2019t relevant to bringing about a change in approach.<\/p>\n

On reflection, our research team could have spent less time running research that only reinforced our early findings that the approach wasn\u2019t achieving the goals of the product teams, and more time helping build a consensus on what the unified approach should be. Although that\u2019s not the \u2018user research\u2019 bit of the job directly, it would have unblocked the effort of the research team earlier, and so is probably the \u2018lead\u2019 bit of the job title.<\/p>\n

A second thing I learned: A service designer has a hard job.<\/strong><\/p>\n

Definitions can be fuzzy, so let\u2019s start with my understanding of what we\u2019re talking about.<\/p>\n

\u2018A service\u2019 is the end-to-end process that lets a user complete a goal, such as renewing their passport, or buying some IT equipment.<\/p>\n

\u2018Service design\u2019 is the practice of defining how that service should work and then making it so.<\/p>\n

So a \u2018service designer\u2019 is the individual who is defining how that service should work and then making it so. But actually doing that, and being a service designer feels incredibly difficult. \u00a0A UI designer can develop a repeatable process for getting their decisions made in real life (and the pixels rarely complain). In contrast service designers don\u2019t (and presumably cannot) have a repeatable process for making people do their job differently, which is required to make any sort of significant change to how most services work. And the people being changed probably will complain, as they may have done their job in the current way for a long time. To achieve any kind of significant change requires a tremendous amount of organisational buy-in and soft skills to make that happen \u2013 which can be difficult to achieve.<\/p>\n

The risk we saw as a research team is that, when blocked from making changes by a lack of buy-in or influence, service designers can be tempted to fall back on further understanding the as-is state, and want to keep running user research in the hope that the situation blocking them will change. This has the potential for conflict when working with an established research team as it generates low impact work for researchers, and hides the reason why change isn\u2019t occurring \u2013 creating the impression that \u2018a lack of research\u2019 is the cause of delay, as opposed to a lack of organisational buy-in, or an appropriate scope being defined for design to occur within.<\/p>\n

One of the tools that helped manage relationships with service designers for us was socialising \u00a0the principles taught to me when I was at PlayStation \u2013 including a clear definition of what is \u2018research\u2019 (planning a study, running a study, drawing conclusions and communicating what was learned), and what is \u2018design\u2019 (having questions that need to be answered by research, deciding what the appropriate action to take is in reaction to what\u2019s been learned). It also impressed upon me the importance of describing the wider context of a round of research \u2013 why is it being run, and what change is going to occur on the back of it. Having a strong understanding of roles and responsibilities feels essential when working with service designers and researchers.<\/p>\n

A third thing I learned: What does a user researcher do?<\/strong><\/p>\n

People will ask user researchers for a lot of things that are not user research, and researchers have to take care not to stray into being a business analyst, product manager or designer. To make good decisions requires the team to understand the organisation, the vision, what is feasible and what good looks like, not just users, and product teams may not have a clear understanding of who can help find each of these out, and will default to asking their user researcher.<\/p>\n

Moving from an organisation where the research team were commissioned to run individual rounds of research, to working in a collaborative multidisciplinary way where roles have a lot more cross-over and grey areas, it took me a while to start to recognise the research questions which weren\u2019t really user research. It reinforced to me the importance of deeply scrutinising the objectives of each round of research we run – stopping, and thinking about what you\u2019re being asked to do, and identify \u201cis it a question that a better understanding of our users will answer\u201d. Related, I saw occurrences of individuals asking for research when they really just need to make a decision. To mitigate this, I\u2019ve lately explored describing the role of user research as primarily to \u201chelp inform decision making\u201d \u2013 to make it clear to teams who are new to working with researchers that user research won\u2019t tell you what to do, and they will still have to make decisions.<\/p>\n

A fourth thing I learned: The importance explaining what you do at all levels<\/strong><\/p>\n

User Research was relatively new at the organisation when I joined, and so we put a lot of effort into trying to explain what it was, demystify it, and make people at all levels of seniority understand how it\u2019s relevant. This can be difficult as people often don\u2019t want to admit \u2013 or are unaware- that they don\u2019t *really* know what it is. Our excellent head of research & design helped reinforce the importance of this, which we did through all staff presentations, open research drop-in sessions, and other initiatives which sum up to \u2018always banging on about research\u2019.<\/p>\n

We saw some issues occur as a result of always talking about research \u2013 a lot of related concepts that aren\u2019t \u201cuser research\u201d (User Centred Design, Service Design, defining what success looks like and measuring if you\u2019re achieving it), often got attributed as user research. Despite these issues, we saw the importance of explaining what we do to people at all levels when organisational changes started to occur \u2013 as we\u2019d put a lot of effort to internally socialising the relevance of our team, I felt relatively safe as senior people had come to the conclusion that our team\u2019s work was important and should continue.<\/p>\n

A fifth thing I learned: Multidisciplinary teams need to be trusted and complete<\/strong><\/p>\n

The ambition when we all started was building autonomous multidisciplinary product teams, in the belief that this would help build better software. And I still have that belief that a good team can build better software in this manner, but I\u2019ve also seen how fragile those teams are, and how easy to disrupt them it can be.<\/p>\n

For many of the iterations of these teams I saw, the decisions about \u201chow should the product work\u201d, \u201cwhat should the product look like\u201d, \u201cwhat things should the product do\u201d, \u201cwhat things should be on the page\u201d were occurring outside of the product teams, which means hand-overs, reviews and changes, and ultimately stopped the product team from being responsible for decisions they have been told to make. This indicates a lack of trust in the work that would be done within product teams from discipline leads, which has worrying implications on the leads.<\/p>\n

We also saw core disciplines not being represented in product teams, which meant that the \u2018process\u2019 of design and development couldn\u2019t occur, and a lot of wasted work by the others in the team. This then creates a spiral of people leaving product teams because the team didn\u2019t have the people within it to deliver, and any shared knowledge is lost. For a research team this taught us a couple of things \u2013 the importance of documentation even when running research in a collaborative way (because shared understanding is lost when people join or leave), and where to focus our efforts \u2013 if a team isn\u2019t going to be able to deliver, research will have low impact for those teams, and our priorities should be elsewhere.<\/p>\n

A sixth thing I learned: Some constraints can be too big to make usable software<\/strong><\/p>\n

For the majority of my time, the development approach for the new website was to understand the concepts and relationships within the organisation, and build HTML representations that map closely to the real objects they describe. As referenced earlier, there are benefits for this approach for sustainability and allowing the civic tech community to use our information to build new things. There was also a hope that it would also create something that was useful to the people who come to our website to complete tasks.<\/p>\n

The implications of this approach for most users became clear to us relatively early on in research, but were slow to lead to an informed critique of the approach and a decision \u201cis this what the organisation really wants from its website\u201d. When we ran research with real users representing the audience of the website, we found that it wasn\u2019t making useful things to people who came to the existing website. The development approach allowed little deviation from the words the organisation used to describe its concepts and processes, or the relationships that the items had in real life when creating structure between pages. But for a complicated and niche subject matter the audience for the website isn\u2019t the people who understand the domain (because there are probably less than ten people who understand the domain in depth, and they all work here). We sought out the real audience for each part of the website and we consistently found that even very specialist users, who use the current website daily, didn\u2019t have the depth of understanding about the language or processes of the organisation to achieve their goals on the HTML representations of the domain. Design is the process of finding solutions within known constraints, but it was evident a robust interpretation layer was necessary between the domain and users, and being denied this layer was one constraint too many for the product teams, who felt prevented from creating things useful to visitors to the website.<\/p>\n

Our discovery research frequently uncovered key journeys that wouldn\u2019t be supported by the HTML representation of the domain. Our usability research reinforced that people couldn\u2019t use the HTML representation of the domain to achieve their goals. But because the goal was to create a HTML representation of the domain, teams felt unable to address those issues and bridge the divide between the domain and people completing useful tasks. The development team has since started to explore changes to its development approach, but I guess the learning I took from this is the same as my previous point \u2013 the previous development approach had different goals to creating useful and usable software for the people who visit the website, and I should have recognised that earlier and thought about the implications of where we should be putting our research time and effort.<\/p>\n

A seventh thing I learned: User Researchers are lovely people<\/strong><\/p>\n

To be fair, I already knew this one. They are also often smart people too, and I really appreciated their thoughts and advice \u2013 anything interesting in my opinions was probably inspired by things they\u2019ve told me, and I\u2019m sure many of them will recognise our conversations in all of these things I learned.<\/p>\n

An eighth thing I learned: Education about the work and doing the work are different things<\/strong><\/p>\n

It seems self evident that doing user research, and teaching people about user research are different things. But there\u2019s lots of grey areas, and it\u2019s easy to fall into the gaps. One that we discussed often was bringing people from outside of the product team into the observation room during usability testing. On the face of it, this seems like a great education opportunity \u2013 seeing what research looks like first hand. However there are a lot of risks, some of which we saw occur first hand.<\/p>\n

Because the guest hasn\u2019t been involved with the product team\u2019s decision that led to that research session, they will find it difficult to take any meaning from the session, and instead the main thing they will take from it is entertainment – seeing what a user looks like in the flesh, and what usability testing looks like. They may come away from that thinking user research is fun, or intriguing. However without seeing the whole process through from kick-off to debrief, the guest won\u2019t come away with any meaningful conclusions about how it fits into the process of design. <\/p>\n

This can be fine, but problems occur when guests are important people, and want to contribute their own observations about what they have seen. Unless a team has been together for a long time, and are mature in their practise and rhythm, these opinions can be distracting for team who won\u2019t know how to incorporate this within the more relevant observations from those closer to understanding \u201cwhat do the team need to know at this point\u201d.<\/p>\n

Some of our researchers used video playback sessions to open up research to a wider group, where they could then moderate the observation experience in addition to the research session. When we were guests at GDS, we were sat at the back \u2013 perhaps physical separation is another appropriate method. Banning guests from the observation room is often unpopular, as no-one wants to hear that they will be a distraction, but perhaps video playback sessions are a way to avoid hard feelings and manage those guests appropriately.<\/p>\n

A ninth thing I learned: Watch out for people using user research as a weapon<\/strong><\/p>\n

I fell for this at least once, and still haven\u2019t worked out all the nuances of how to avoid this. Because research finds \u2018truth\u2019, there is a risk of it being commissioned to prove a point \u2013 particularly when someone wants to prove a colleague wrong.<\/p>\n

One of the more obvious tells is when research is commissioned on someone else\u2019s work \u2013 unless they have the ability to make changes to it, the impact of your research is likely to be nothing. Especially if the commissioner doesn\u2019t have any agreement that evidence is going to inform decision making (\u201cyou can\u2019t reason someone out of a position they didn\u2019t reason themselves into\u201d).<\/p>\n

But it\u2019s not always that obvious, and there are a lot of grey areas, which I haven\u2019t got my head around yet. Ultimately the \u2018point\u2019 of a user research team is to help individuals and organisations make better decisions, and I can see that commissioning research to help inform viewpoints is a step towards evidence based decision making. But it has a lot of internal political risks also. I have no advice other than be careful\u2026<\/p>\n

A tenth thing I learned: Only you can be responsible for doing good work<\/strong><\/p>\n

A popular ethos in a lot of public sector research is in collaborative research \u2013 active involvement of the whole team in the research process to increase their understanding of the situation for real users. However what the rest of the team will lack is an understanding of the rigour that should be going into designing and running studies and drawing conclusions, and how to appropriately caveat results to ensure that the results are communicated accurately, and are as \u2018true\u2019 as possible.<\/p>\n

You can imagine a scale of rigour from \u201cjust making it up\u201d, to \u201ca peer reviewed, repeatable academic study\u201d, and plot any study on that scale. The lowest bar for a researcher working in industry is \u201cmy colleagues will believe me\u201d. But that is far lower than the bar we should be setting of \u201cthis meets my professional and ethical standards\u201d. The risk is that no-one will be checking it meets your professional standards, and the only checkpoint is whether your colleagues believe you. And because decisions made based on research can often take years to come to fruition, there is very low accountability for poor quality research. So being a good user researcher requires commitment and honesty, far beyond what anyone other than yourself will be able to assess.<\/p>\n

A thing I found helpful during my time working in the public sector was to remain a member of the games user research community, and be exposed to their discussions, for example via their discord<\/a>. Ensuring that data is accurately gathered and honestly reported is a common discussion topic, and being exposed to the professional expertise of other researchers helped set my standard appropriately.<\/p>\n

The risk when running collaborative research with a wider team who are not researchers is that they won\u2019t understand or be able to apply appropriate rigour to identify \u2018safe\u2019 conclusions and \u2018unsafe\u2019 conclusions. One of the tools our researchers have applied is education about the \u2018synthesis\u2019 stage \u2013 that gap between data collection and debriefing, where a researcher goes away and thinks about the results properly, and translates the team\u2019s early thoughts into results we are confident in. Ensuring that teams we work with understand and see the value in that stage is essential.<\/p>\n

The eleventh thing I learned: \u201cit depends\u201d<\/strong><\/p>\n

There\u2019s a lot of trendy catchphrases and opinions in the wider user centred design space. I have often been guilty of it. Some you may hear include\u2026<\/p>\n