Covering the full trial life cycle
It is important for the steering group to be able to monitor trial performance throughout the different stages. This module sets out some key questions that the steering group, together with those trialling an innovation, should seek answers to. This will allow the steering group to introduce stage gates at the end of each phase to ensure trials deliver in line with expectations.
Feasibility / Technology
Prepare
Work to ensure the trial is focused and refined enough to be successful (balance ambition and pragmatism)
- Is the tech actually ready to trial (or does it still require extensive development)?
- What barriers is the technology likely to have to overcome for installation? (Does it require additional space, tech, infrastructure, personnel, or approvals to enable trialling?)
- Ensure the trial is collecting sufficient data to measure trial success against objectives, and is compliant with data regulations
- Understand what regulations need to be met before a solution can be installed in the station
Running
Trialling of technology and installation at the station.
- Ensure users (passengers and station staff) are engaging meaningfully with the tech/trial
- Collect sufficient data to ensure successful Monitoring and Evaluation
- Ensure the SME has met all relevant requirements before the trial begins
Analyise
Analyse data to understand how well the innovation is performing success against trial objectives
- Are there any unintended consequences of the tech?
- Are there learnings about the innovation itself in addition to learnings about the impact of the innovation?
Evaluate
Use the analysed data to evaluate how the innovation has performed (technically) against the objectives and success factors determined in the preparation phase
- Evaluate whether the innovation is suitable for more extensive deployment at the station, or other stations.
Be ambitious:
The SIZ is there to try out new things. Test bold ideas and take risks to make the most of the opportunity to learn about what changes have the most positive impact. Not every trial has to go the procurement route afterwards…
Desirability / Human-Centred
Prepare
Understand which questions exist around the innovation in use at the station.
- Determine who the front-and back-end users are that need to be consulted, and recruit participants. • Do the questions differ across different user groups?
- Consider the best approaches reach out to the users to involve in trialling the innovation.
Running
Collect data in the form of feedback from specified users. Choose an appropriate approach to quantitative and/ or qualitative data collection.
Analyse
Understand the positive and negative experiences of the solutions in use, the impact that is therefore likely to have on the front- and back-end users. Applying quantitative or qualitative data analysis. approaches, in line with the data collection method.
Evaluate
Articulate well-balanced answers informed by the learnings from the analysis to answer the research questions posed at the start of the process.
- What was the impact of the innovation against the challenge area? Did it (partway) achieve the positive change that was envisioned?
- Understand what changes may need to be made to improve the impact of the innovation.
Viability / Commerical
Prepare
Prepare a business model for each innovation, focusing on viability of the innovation once it is deployed at the station.
- Who are the right teams/individuals to review the commercial viability of the innovation at the station?
Running
Work with NR’s procurement teams to review the potential routes to market.
- Explore the commercial offerings of the solution and how it can align with your model
- Explore what budget is available post trial.
Analyse
How have the commercial exploration discussions gone?
- Has the SME been able to find out enough information to assess if the model would be viable for both parties?
Evaluate
Evaluate the learnings gathered around the business model and potential routes to market.
- What is the overall verdict of experts in the field?
- What are barriers identified that need to be overcome for the solution to be ready for deployment in terms of viability?
BTM Learning
- Stations may have set assessment processes to measure performance. Network Rail has a scorecard for which the targets are updated yearly. Understanding these existing structures can inform assessment of trial innovations., Within SIZ with each SME cohort challenge setting has become more closely aligned with scorecard targets with the ambition to demonstrate how trialled solutions can contribute to achieving improved station performance.
- Make sure that each SME has a meeting with the station to detail installation requirements so NR can create a cost and installation plan based on the station. We discovered that some installations required a possession which had not been factored in.
- Recruiting trial participants was harder than some of the SMEs anticipated. We saw huge variations of the breadth and numbers of participants within their trial plan compared to what they were actually able to recruit. Make sure that there are no assumptions in the trial plan about the availability of required participants e.g. a representative organisation will simply supply these people.
- One of the SMEs struggled to recruit their required participants to complete their user testing during the trial highlighting the need to kick-start recruitment as early as possible and identify before this any support they SME will need with this. Since year 2, the recruitment exercise has been moved to the trial proposal phase, with the innovation facilitator supporting the SMEs.
- Fortnightly meetings for the core project team between NR and the SMEs worked well and enabled any issues to be solved in a timely manner.
- By year 3, we reflected on the time constraints imposed on the trials. Due to budget constraints, time constraints arise. We believe more time will enable stakeholders to have a greater input, and make more data sets available to measure the trials against, leading the trials to provide greater value.