Overview of output files
This section explains the meaning of the output files.
Output - patient pathway (Simmer)
Output - logs
Full simulation logs are saved if flag_savedata_fulllogs <- TRUE in config.R.
These will be large if used for very large simulations (e.g. high number of replications / volume of arrivals / attributes) so if not directly needed at this granularity, could be turned off.
| File | Description | |
|---|---|---|
| log_arrivals | A log of arrivals (demand). Both ambulance and direct. Each row is an arrival. | |
| log_resources | An event log of resource use. Both double staffed vehicles and ED. A new row signals a timestamp in which either the queue, occupancy or capacity of a resource changed. | |
| log_attributes_w | A log of arrivals and their various characteristics and notable timestamps. Attributes saved are defined in the trajectory.R specification. |
Output - log of arrivals
Variables in log_arrivals.xlsx.
| File | Description | |
|---|---|---|
| name | Unique patient identifier. | |
| start_time | Start of activity. | |
| end_time | End of activity (model exit) | |
| activity_time | Difference between start and end time | |
| finished | Flag on whether activity had finished by the end of the simulation | |
| replication | The model run replication number |
Output - log of resources
Variables in log_resources.xlsx.
| Variable | Description | |
|---|---|---|
| resource | Resource (server) type. | |
| time | Timestamp marking change in utilisation profile from previous row's characteristics | |
| server | Number patients in server | |
| queue | Number patients in queue for that server | |
| capacity | Capacity of that server | |
| queue_size | Maximum allowed queue size of that server (#NUM means infinite) | |
| system | Number patients in server system (in queue or in server itself) | |
| limit | Maximum allowed system (queue+server) size (#NUM means infinite) | |
| replication | The model run replication number | |
| id | n/a |
Output - log of attributes
Variables in log_attributes_w.xlsx.
| Variable | Description |
|---|---|
| name | Unique patient identifier (prefix "PatientD_" if Direct/Walk-In) |
| replication | The model replication number |
| id | nan |
| Cat | The ambulance category (1,2,3,4). Will be blank for Direct/Walk-In. |
| Convey | The care model (1=S&C ; 2=S&T; 3=H&T). Will be blank for Direct/Walk-In. |
| T_timetoscene | The sampled time to scene interval (seconds) |
| T_timeatscene | The sampled time at scene interval (seconds) |
| T_timetosite | The sampled time to site interval (seconds) |
| T_unavprehand | The sampled unavailable pre-handover time interval (seconds) |
| T_timetoclear | The sampled time to clear interval (seconds) |
| t_Rclockstart | The timestamp of response time clock start (here same as arrival) |
| n_resources_allocate | The sampled number of resources to allocate. |
| n_resources_arrive | The sampled number of resources to arrive at site. |
| t_ambulanceseized | The timestamp at which an ambulance could be seized / allocated. |
| t_hospitalarrival | The timestamp of hospital arrival. |
| Acuity | The sampled acuity (1=highest/resus ; 5 = lowest/minor) |
| LoS | Defined time in A&E department. |
| t_EDhandover | The timestamp of ED handover |
| t_Rclockstop | The timestamp of response time clock stop / arrival at scene |
| Done | Indication of whether the activity is done by the endo f the simulation and how (1=S&C ; 2=S&T ; 3=H&T; 0=Direct; blank means not finished) |
| t_JCTend | The timestamp of the job cycle time end (ambulance perspective) |
| ConveyDestED | The conveyance destination |
| t_ambclear | The timestamp the ambulance cleared |
| t_EDclockstop | The timestamp of ED clock stop (end of time in A&E) |
| T_allocate | The resulting time to allocate interval (seconds) = t_ambulanceseized - t_Rclockstart |
| T_prehandover | The resulting time to handover interval (seconds) = t_Edhandover - t_hospitalarrival |
| T_JCT | The resulting total JCT time interval (seconds) = t_JCTend - t_Rclockstart (this differs from a more classic definition that excludes time from call connect to allocate as till that point no resource has been on the job) |
| Kp_Tresponsetime | The resulting responde time interval (seconds) = t_Rclockstop - t_Rclockstart |
| T_ED | The time in ED interval (seconds) = t_Edclockstop - t_Edhandover |
Output - KPI files
Overall (simulation time) KPIs are saved if flag_savedata_overallKPIs <- TRUE in config.R.
Step-wise (by steps of simulation time) KPIs are saved if flag_savedata_stepwiseKPIs <- TRUE in config.R.
Contrary to the full logs, these files save more post-processed KPIs (though all resulting from those two logs).
KPIs include:
- Response times (Cat1, 2, 3, 4) - mean and 90th percentile
- Handover delays 30+ (%)
- Handover delays 60+ (%)
- Total time lost to handover delays (TLAHD)
- Total job cycle time (JCT)
- Double crewed ambulance (DCA) time - mean, sd
- Time to allocate - mean, sd
- Call queue (queueing for an ambulance)
- Site queue (queueing for ED)
- Resource utilisation (%) - mean, sd
Replication or batch: Some files give results "by replication" while others summarise these further "across replications" ("batch"). This is indicated in the table below.
If batch - replication aggregate or flat: In some cases, this "batch" KPI is derived by considering all arrivals regardless of replication ("flat"). In others, the batch KPI is found by first calculating the KPI per replication and then finding the mean and standard deviation of this KPI among replications ("aggregate").
Stepwise: Some KPIs are calculated for different steps (e.g. days, time windows) of the simulation window while other times they are calculated for the overall simulation time.
Disaggregations: Where certain disaggregations were used, this is indicated (e.g. category for response times ; care model for time to allocate; resource type for utilisation; etc)
An overview of the meaning of KPI output files is given below (navigate rightwards in the table to see all columns).
| Filename | Description | KPI theme | KPIs | Statistical measures | Replication or batch | If batch - replication aggregate or flat | Stepwise | Disaggregations | Comments |
|---|---|---|---|---|---|---|---|---|---|
| row.xlsx | Log of what-if levers in this scenario | nan | nan | nan | Batch | nan | nan | nan | nan |
| df_scenarios.xlsx | Log of what-if levels across scenarios | nan | nan | nan | Scenario | nan | nan | nan | nan |
| KPI_overall_DCAtime.xlsx | Double-crewed ambulance time | DCA time | DCA time | median, mean, sd, n | Batch | Flat | nan | nan | nan |
| KPI_overall_HO.xlsx | Handover delay KPIs | Handover delays | Handover delays | HO30+, HO60+, TLAHD, n_ambu | Batch | Replication aggregate - mean, sd | nan | nan | nan |
| KPI_overall_HO_deciles.xlsx | Handover delay KPIs | Handover delays | Handover delays | Deciles | Batch | Flat | nan | nan | nan |
| KPI_overall_JCT.xlsx | Job cycle time | JCT | JCT | median, mean, sd, n | Batch | Flat | nan | nan | nan |
| KPI_overall_JCT_conv.xlsx | Job cycle time | JCT | JCT | median, mean, sd, n | Batch | Flat | nan | conveyance | nan |
| KPI_overall_val_cat.xlsx | Average time to allocation; time for prehandover; job cycle time; response time, by category | JCT components | T_allocate; T_prehandover; T_JCT; T_responsetime | mean, median | Batch | Flat | nan | category | nan |
| KPI_overall_val_catconv.xlsx | Average time to allocation; time for prehandover; job cycle time; response time, by category and conveyance | JCT components | T_allocate; T_prehandover; T_JCT; T_responsetime | mean, median | Batch | Flat | nan | category; conveyance | nan |
| KPI_overall_val_conv.xlsx | Average time to allocation; time for prehandover; job cycle time; response time, by conveyance | JCT components | T_allocate; T_prehandover; T_JCT; T_responsetime | mean, median | Batch | Flat | nan | conveyance | nan |
| KPI_step_CallStack.xlsx | Call Stack by replication and simulation timestep interval | Call Stack | Call Queue | mean | Replication | nan | Yes | nan | nan |
| KPI_step_HO.xlsx | Handover delay KPIs by replication and timestep interval | Handover delays | Handover delays | HO30+, HO60+, TLAHD, n_ambu | Replication | nan | Yes | nan | nan |
| KPI_step_RT.xlsx | Response time KPIs by replication, timestep interval and category | Response times | Response times | median, 90%ile, mean | Replication | nan | Yes | category | nan |
| KPI_step_SiteQueue.xlsx | Site Queue by replication and simulation timestep interval | Site queue | Site Queue | mean | Replication | nan | Yes | nan | nan |
| KPIbatch_overall_RT.xlsx | Overall response time | Response times | Response time | median, 90%ile, mean | Batch | Replication aggregate - mean | nan | nan | nan |
| KPIbatch_overall_RTcat.xlsx | Response time by category | Response times | Response time | median, 90%ile, mean | Batch | Replication aggregate - mean, sd | nan | category | nan |
| KPIbatch_overall_RU.xlsx | Resource utilisation | Resource utilisation | Utilisation | Utilisation | Batch | Replication aggregate - mean, sd | nan | resource type | nan |
| Queue | Queue | ||||||||
| KPIbatch_overall_val.xlsx | Batch-level average time to allocation; time for prehandover; job cycle time; response time | JCT components | T_allocate; T_prehandover; T_JCT; T_responsetime | mean, median | Batch | Replication aggregate - mean, sd | nan | nan | nan |
| KPIflat_LoS | Time in department by category | Time in A&E department | Time in A&E department | mean | Batch | Flat - mean | nan | category | nan |
| KPIbatch_LoS | Time in department by category by replication | Time in A&E department | Time in A&E department | mean | Replication | nan | nan | category | nan |
| KPIrep_overall_HO | Handover delay KPIs by replication | Handover delays | Handover delays | HO30+, HO60+, TLAHD, n_ambu | Replication | nan | nan | nan | nan |
| KPIrep_overall_HT | Overall response time by replication | Response times | Response time | median, 90%ile, mean | Replication | nan | nan | nan | nan |
| KPIrep_overall_RT_cat | Response time by category and replication | Response times | Response time | median, 90%ile, mean | Replication | nan | nan | category | nan |
| KPIrep_overall_RU | Resource utilisation by replication | Resource utilisation | Utilisation | Utilisation | Replication | nan | nan | resource type | nan |
| Queue | Queue | ||||||||
| KPIrep_overall_val_cat | Average time to allocation; time for prehandover; job cycle time; response time, by category and replication | JCT components | T_allocate; T_prehandover; T_JCT; T_responsetime | mean, median | Replication | nan | nan | category | nan |
| parameterlistaudit.xlsx | An Rdata file keeping a log of most parameters (globals, user) used in the model rul | nan | nan | nan | nan | nan | nan | nan | nan |
Output - plot files
A set of plots related to the KPIs are saved if flag_saveplots <- TRUE in config.R.
Some plots may, depending on the nature of the scenario, benefit from different scaling of presentation. When a run ends, all plots are kept in memory as plot objects in a structure called plot_list. These can be further adapted with ggplot as needed, or for instance saved as vector files (.svg) for image processing.
Alternatively, though we generate these plots for ease, all of them can be generated independently by the user from one of the three full logs.
An overview of the plots is given below.
| Filename | Description | Derived via |
|---|---|---|
| plot_arri_cat_CM | Arrivals over time, per category and conveyance. Arrivals for walk-ins (99). | log of attributes |
| plot_arri_cat_DnC | Arrivals over time for three replications and by category. Demand input in black, supply schedule in red. | log of attributes |
| plot_Kp_HO_bp | Handover KPIs stepwise over simulation time. Boxplots show variation among replications. Dashed horizontal lines will show reference levels given by user. | log of attributes |
| plot_Kp_HO_bp_y0 | As 'plot_Kp_HO_bp' but starting at y=0 | log of attributes |
| plot_Kp_HO_bp_rep | As 'plot_Kp_HO_bp' but by replication (up to 3) | log of attributes |
| plot_Kp_RT90_bp_d14 | 90th percentile of response time by category, in the timestep. Boxplots show variation among replications. | log of attributes |
| plot_Kp_RT90_bp_d14 | As 'plot_Kp_RT90_bp_d14' but vertically stacked. | log of attributes |
| plot_Kp_RTmean_bp_d14 | Mean of response time by category, in the timestep. Boxplots show variation among replications. | log of attributes |
| plot_Kp_Rtmean_bp_ref_freey_d14 | As 'plot_Kp_Rtmean_bp' but vertically stacked. Dashed horizontal lines will show reference levels given by user. | log of attributes |
| plot_Kp_Rtmean_bp_ref_freey0_d14 | As 'plot_Kp_Rtmean_bp_ref_freey_d14' but starting at y=0. | log of attributes |
| plot_RU_instant_ | Instantaneous utilisation of resources (ambulance, A&E bay). Server capacity as dashed blue line. Server use as full blue line. Queues shown as filled red lines. Multiple traces correspond to multiple replications. | log of resources |
| plot_RU_KPI | Overall utilisation of resources (ambulance, A&E bay), % | log of resources |
| p_KPI_cq_1 | Average calls queued per time step. Boxplots show variation among replications. Time step as 6 hours. | log of attributes |
| p_KPI_sq_1_day | Average ambulances queued at ED per time step. Boxplots show variation among replications. Time step as 24 hours. | log of attributes |