It is important for the steering group to be able to monitor trial performance throughout the different stages. This module sets out some key questions that the steering group, together with those trialling an innovation, should seek answers to. This will allow the steering group to introduce stage gates at the end of each phase to ensure trials deliver in line with expectations.
Feasibility / Technology
Prepare
Work to ensure the trial is focused and refined enough to be successful (balance ambition and pragmatism)
- Is the tech actually ready to trial (or does it still require extensive development)?
- What barriers is the technology likely to have to overcome for installation? (Does it require additional space, tech, infrastructure, personnel, or approvals to enable trialling?)
- Ensure the trial is collecting sufficient data to measure trial success against objectives, and is compliant with data regulations
- Understand what regulations need to be met before a solution can be installed in the station
Running
Trialling of technology and installation at the station.
- Ensure users (passengers and station staff) are engaging meaningfully with the tech/trial
- Collect sufficient data to ensure successful Monitoring and Evaluation
- Ensure the SME has met all relevant requirements before the trial begins
Analyise
Analyse data to understand how well the innovation is performing success against trial objectives
- Are there any unintended consequences of the tech?
- Are there learnings about the innovation itself in addition to learnings about the impact of the innovation?
Evaluate
Use the analysed data to evaluate how the innovation has performed (technically) against the objectives and success factors determined in the preparation phase
- Evaluate whether the innovation is suitable for more extensive deployment at the station, or other stations.
Be ambitious:
The SIZ is there to try out new things. Test bold ideas and take risks to make the most of the opportunity to learn about what changes have the most positive impact. Not every trial has to go the procurement route afterwards…
Desirability / Human-Centred
Prepare
Understand which questions exist around the innovation in use at the station.
- Determine who the front-and back-end users are that need to be consulted, and recruit participants. • Do the questions differ across different user groups?
- Consider the best approaches reach out to the users to involve in trialling the innovation.
Running
Collect data in the form of feedback from specified users. Choose an appropriate approach to quantitative and/ or qualitative data collection.
Analyse
Understand the positive and negative experiences of the solutions in use, the impact that is therefore likely to have on the front- and back-end users. Applying quantitative or qualitative data analysis. approaches, in line with the data collection method.
Evaluate
Articulate well-balanced answers informed by the learnings from the analysis to answer the research questions posed at the start of the process.
- What was the impact of the innovation against the challenge area? Did it (partway) achieve the positive change that was envisioned?
- Understand what changes may need to be made to improve the impact of the innovation.
Viability / Commerical
Prepare
Prepare a business model for each innovation, focusing on viability of the innovation once it is deployed at the station.
- Who are the right teams/individuals to review the commercial viability of the innovation at the station?
Running
Work with NR’s procurement teams to review the potential routes to market.
- Explore the commercial offerings of the solution and how it can align with your model
- Explore what budget is available post trial.
Analyse
How have the commercial exploration discussions gone?
- Has the SME been able to find out enough information to assess if the model would be viable for both parties?
Evaluate
Evaluate the learnings gathered around the business model and potential routes to market.
- What is the overall verdict of experts in the field?
- What are barriers identified that need to be overcome for the solution to be ready for deployment in terms of viability?
BTM Learning
- Stations may have set assessment processes to measure performance. Network Rail has a scorecard for which the targets are updated yearly. Understanding these existing structures can inform assessment of trial innovations., Within SIZ with each SME cohort challenge setting has become more closely aligned with scorecard targets with the ambition to demonstrate how trialled solutions can contribute to achieving improved station performance.
- Make sure that each SME has a meeting with the station to detail installation requirements so NR can create a cost and installation plan based on the station. We discovered that some installations required a possession which had not been factored in.
- Recruiting trial participants was harder than some of the SMEs anticipated. We saw huge variations of the breadth and numbers of participants within their trial plan compared to what they were actually able to recruit. Make sure that there are no assumptions in the trial plan about the availability of required participants e.g. a representative organisation will simply supply these people.
- One of the SMEs struggled to recruit their required participants to complete their user testing during the trial highlighting the need to kick-start recruitment as early as possible and identify before this any support they SME will need with this. Since year 2, the recruitment exercise has been moved to the trial proposal phase, with the innovation facilitator supporting the SMEs.
- Fortnightly meetings for the core project team between NR and the SMEs worked well and enabled any issues to be solved in a timely manner.
- By year 3, we reflected on the time constraints imposed on the trials. Due to budget constraints, time constraints arise. We believe more time will enable stakeholders to have a greater input, and make more data sets available to measure the trials against, leading the trials to provide greater value.
Keeping a holistic approach is essential to understanding the impact of the innovations that are being tested. It allows the team to understand how well each innovation performs in the station context and whether it meets the needs of stakeholders across the technological, commercial and human-centred perspectives. This will help to evaluate the impact of the innovations tested in the Zone.
Feasibility / Technology
A technology perspective assesses feasibility, and gives you insight into the efficiency of the solution.
Two main areas which need to be considered to assess the feasibility of the trial are:
- Technology readiness
- Station readiness
Technology readiness
- Is the technology ready to be trialled, or does it still require extensive development?
- It is important to be realistic about the desired outcome for the technology from trialling in the station reflecting how well the technology is developed (pre-trial) and what success looks like (post-trial).
Station readiness
The success of the trial is party down to ensuring that the station is ready for the solution to be introduced, meeting relevant requirements.
Reviewing these points as part of taking a technology-centred perspective will
- Increase knowledge of new technologies and how they can feasibly be trialled
- Grow your understanding of how stations can adapt to incorporate new innovations
As the technological feasibility is understood, it is important to ascertain whether the innovation is responding to a clear need.
Desirability / human centred
A human-centred perspective assesses desirability, and gives you insight into the effectiveness of the solution.
People are a major factor in the success of innovation. A ‘human-centred’ perspective looks at understanding the impact and experience of the solution in use, including:
- The range of potential users
- The different contexts of use (use cases)
- Use in the ‘here and now’ and over time
- Identifying the different needs of those using the solution as end-users, and those running and maintaining the solutions for those end-users
Taking a human-centred perspective will:
- The range of potential users
- The different contexts of use (use cases)
- Use in the ‘here and now’ and over time
- Identifying the different needs of those using the solution as end-users, and those running and maintaining the solutions for those end-users
Viability / Commercial
A commercial perspective assesses viability, and gives you insight into the affordability of the solution.
Throughout the trials, not only is it important to test that the solutions will feasibly work in the station context, but it is also an opportunity to explore how the commercial model would align with your own business and procurement routes.
Things to consider
- How does the trial plan align with your strategy?
- Does the commercial model align with how things usually work, if not, what other options are there?
- Are there possible pathways identified that can be integrated that could be integrated with the funding model of the SIZ?
- Have procurement routes been made clear and does the supplier meet the requirements?
BTM Learning
- It can be difficult for station staff to give time and energy to reflect on the impact of the trialled solutions due to their busy jobs.
- Due diligence is key to ensure innovations can be trialled: We learned from the first round that we needed a better understanding of fire, safety and accessibility requirements to ensure we would not experience delays in the second round of trialling.
- Make sure to be clear that the participants are taking part in a trial, and that their feedback is very welcome. One of the SMEs received very critical feedback on their solution from a participant. When reflecting on this incident, the SME shared that they felt they could have been better in communicating what the purpose of the trial was..
- It is important to consider all those who may benefit from or are impacted in some other way by a trialled solution.Engaging with as many of these stakeholders allows for a full evaluation of the solution. Including understanding the most impactful use cases and the integration of the solution into organisational structures and procedures.
- One of the solutions we trialled BTM received good feedback and station staff were engaged. When discussions were turned towards workloads and resources, it became clear that this was a much stronger barrier than the station’s staff willingness to adopt the solution.
Reviewing these points will allow you to assess the viability of the solution beyond the trial
Monitoring and Evaluation describes the action of collecting data and examining the relevance, effectiveness, efficiency, and impact of activities in the light of specified objectives, outcomes and impact by identifying what did or did not work during a trial.
Producing a good quality evaluation will lead to improved innovation products, both from gaining more useful insights about the product/service being trialled and with regards to making future innovation testing more useful
Staying aware
Collect an ample amount of robust data to extract useful findings from a trial. However, this data will not be useful unless you are able to extract meaning from it.
Define success
Through trialling, you aim to better understand the impact of innovations on the users and the system, and how effective the innovations are at achieving it. By taking the approach of continuous learning and regular feedback, you ensure the views of front- and back-end users of the solutions that you are trialling are heard. This will help the solutions be human centred and allow you to increase understanding of people’s needs at the station. Remember, this applies to station staff, maintenance & operations teams, and servicing suppliers as much as passengers and other station visitors.
Prior to running a trial, define metrics which showcase a trial’s success. These need to be carefully considered to ensure they are achievable whilst also resulting in enough evidence to gain real insight from the trial. Conduct a needs assessment to determine the triallist’s capability to capture the data appropriately. Also make sure that data collected can describe the impact of the solution in a way that is meaningful to the rail industry (aligns with their measures of success).
Define these prior to trial commitments to allow comparison of trial results with initial expectations and adjust future practices for the discrepancy. Examples are:
- The number of participants included in a trial
- The amount of data produced, or people engaged, during the trial
- The desired response of trial subjects and testers (e.g., the public)
- Data which supports your existing commercial business model
- Data which supports the aims and objectives of key industry stakeholders
- Consider the balance between what different stakeholders want to monitor
- Set KPI’s and use benchmarks to do comparisons
Monitoring and Evaluation Outputs
Outputs will vary depending on what is being tested. Whilst quantified impacts are often the easiest to understand sometimes it will only be possible to collect qualitative data. The key outputs you should attempt to gain from monitoring and evaluation are views of:
- How well the trial went and what could have been improved,
- What barriers did the trial face and how were these overcome?
- Did any additional changes need to be made to the product/service being tested to conduct the trial?
- Was the testing rigorous enough to form a solid opinion from trial results?
- What were the positive/negative views of the product or service being tested?
- Did users or results match initial expectations or previous trial results?
The outcomes of a trial may increase confidence in buyers of innovations that the solutions have been tested in the station. Understanding existing monitoring and evaluation processes to identify opportunities to participate in these as well as build on them. Furthermore, think about how monitoring of the trial can inform the business case for the solution. We therefore recommend you develop a template for a ‘formal’ trial finishing: A post-trial impact assessment and exploitation plan.
When completed by the triallist, this plan identifies next steps for them to take to continue their journey of becoming an industry supplier. The template builds strongly on on existing impact measures forms used in industry, such as ‘score cards’ and Transport Focus’ discontinued ‘National Rail Passenger Survey.
BTM Learning
- In year 1, impact assessments were completed for the individual SME trials, but as well as assessing individual trials it is beneficial to look at the wider impact of the programme.
- For this Key Performance Indicators and Critical Success Factors for the programme were defined- this was a tricky process and we will continue to review and refine these throughout the programme.
- The continuity of station staff supporting the trials from feasibility through to the trial phase was a success. There was a positive cultural shift amongst BTM staff as they were engaging with the trials and the programme. This positive cultural shift continued throughout year 2 of the programme. Introducing the Testbed managerwith their industry knowledge and station operational experience enabled interest and positivity towards the trials to increase.
- At the end of the trials, we found that some SME trial end-reports did not clearly inform stakeholders of the SMEs learnings, including the value to the SME generated by trialling, next steps for their innovation and the impact that stakeholders achieved. A formal close out at the end of the trial period is necessary for both SIZ and the SMEs to show the value and identify next steps. An exploitation plan post-trial is essential.
- From year 3, we have started to find the need for impact assessment and exploitation plans increasingly important for the success of our SIZ at BTM, to show the benefits of a testbed and trials in preparing innovations for the market The SIZ is operating well in terms of setting up and running the trials, and so we can increase focus on how the SIZ continues beyond the trials. Our learning is that we use industry language to allow triallists to write up their trial story so that industry can link it to their ambitions and requirements. Knowing what is needed for a trialist’s next step will help collect data to support this.
