Considerations for monitoring trials
Monitoring and Evaluation describes the action of collecting data and examining the relevance, effectiveness, efficiency, and impact of activities in the light of specified objectives, outcomes and impact by identifying what did or did not work during a trial.
Producing a good quality evaluation will lead to improved innovation products, both from gaining more useful insights about the product/service being trialled and with regards to making future innovation testing more useful
Staying aware
Collect an ample amount of robust data to extract useful findings from a trial. However, this data will not be useful unless you are able to extract meaning from it.
Define success
Through trialling, you aim to better understand the impact of innovations on the users and the system, and how effective the innovations are at achieving it. By taking the approach of continuous learning and regular feedback, you ensure the views of front- and back-end users of the solutions that you are trialling are heard. This will help the solutions be human centred and allow you to increase understanding of people’s needs at the station. Remember, this applies to station staff, maintenance & operations teams, and servicing suppliers as much as passengers and other station visitors.​
Prior to running a trial, define metrics which showcase a trial’s success. These need to be carefully considered to ensure they are achievable whilst also resulting in enough evidence to gain real insight from the trial. Conduct a needs assessment to determine the triallist’s capability to capture the data appropriately. Also make sure that data collected can describe the impact of the solution in a way that is meaningful to the rail industry (aligns with their measures of success).

Define these prior to trial commitments to allow comparison of trial results with initial expectations and adjust future practices for the discrepancy. Examples are:
- The number of participants included in a trial
- The amount of data produced, or people engaged, during the trial
- The desired response of trial subjects and testers (e.g., the public)
- Data which supports your existing commercial business model
- Data which supports the aims and objectives of key industry stakeholders
- Consider the balance between what different stakeholders want to monitor
- Set KPI’s and use benchmarks to do comparisons
Monitoring and Evaluation Outputs
Outputs will vary depending on what is being tested. Whilst quantified impacts are often the easiest to understand sometimes it will only be possible to collect qualitative data. The key outputs you should attempt to gain from monitoring and evaluation are views of:

- How well the trial went and what could have been improved,
- What barriers did the trial face and how were these overcome?
- Did any additional changes need to be made to the product/service being tested to conduct the trial?
- Was the testing rigorous enough to form a solid opinion from trial results?
- What were the positive/negative views of the product or service being tested?
- Did users or results match initial expectations or previous trial results?
The outcomes of a trial may increase confidence in buyers of innovations that the solutions have been tested in the station. Understanding existing monitoring and evaluation processes to identify opportunities to participate in these as well as build on them. Furthermore, think about how monitoring of the trial can inform the business case for the solution. We therefore recommend you develop a template for a ‘formal’ trial finishing: A post-trial impact assessment and exploitation plan.
When completed by the triallist, this plan identifies next steps for them to take to continue their journey of becoming an industry supplier. The template builds strongly on on existing impact measures forms used in industry, such as ‘score cards’ and Transport Focus’ discontinued ‘National Rail Passenger Survey.
BTM Learning
- In year 1, impact assessments were completed for the individual SME trials, but as well as assessing individual trials it is beneficial to look at the wider impact of the programme.
- For this Key Performance Indicators and Critical Success Factors for the programme were defined- this was a tricky process and we will continue to review and refine these throughout the programme.
- The continuity of station staff supporting the trials from feasibility through to the trial phase was a success. There was a positive cultural shift amongst BTM staff as they were engaging with the trials and the programme. This positive cultural shift continued throughout year 2 of the programme. Introducing the Testbed managerwith their industry knowledge and station operational experience enabled interest and positivity towards the trials to increase.
- At the end of the trials, we found that some SME trial end-reports did not clearly inform stakeholders of the SMEs learnings, including the value to the SME generated by trialling, next steps for their innovation and the impact that stakeholders achieved. A formal close out at the end of the trial period is necessary for both SIZ and the SMEs to show the value and identify next steps. An exploitation plan post-trial is essential.
- From year 3, we have started to find the need for impact assessment and exploitation plans increasingly important for the success of our SIZ at BTM, to show the benefits of a testbed and trials in preparing innovations for the market The SIZ is operating well in terms of setting up and running the trials, and so we can increase focus on how the SIZ continues beyond the trials. Our learning is that we use industry language to allow triallists to write up their trial story so that industry can link it to their ambitions and requirements. Knowing what is needed for a trialist’s next step will help collect data to support this.