Evaluating remote post-mortem veterinary meat inspections on pig carcasses using pre-recorded video material

Abstract

Background

Official meat inspections at small-scale slaughterhouses and game-handling establishments in geographically remote areas place a heavy burden on the meat-producing food business operators. By performing meat inspections remotely using live-streamed video, instead of on-site, the official control could meet the goals of sustainability, resilience and logistics. We investigated the agreement between the two approaches at pig slaughter. Two official veterinarians (OVs) inspected 400 pig carcasses at a Swedish slaughterhouse, with each pig being inspected on site by one OV and remotely by the other. After a period of 3 to 6 months, video recordings of the remote inspections were assessed again by the same OVs, thus enabling direct comparisons of previous on-site inspections and renewed video-based inspections within the same OV.

Results

Agreement across 22 finding codes was generally very high for both OVs. In all but one case (whether to fully condemn a carcass), for both OVs, Prevalence-Adjusted Bias-Adjusted kappa was well above 0.8, indicating ‘almost perfect agreement’.

Conclusions

This study supports earlier findings that reliable post-mortem inspections can be performed using video, and indicates higher agreement between remote and on-site inspections if the same OV performs both.

Background

All animals processed at slaughterhouses and game-handling establishments designated to the common market in the European Union (EU) must undergo regulated on-site controls according to EU regulations [1]. These inspections are carried out under the supervision or responsibility of an official veterinarian (OV). Flexibility in the regulations creates the opportunity for official auxiliaries (OA) and specifically designated control personnel to perform certain tasks. Rules in force demanding ante- and post-mortem inspections (AMI, PMI) to be performed on site for each animal, even in low-capacity establishments in remote areas, create large problems from a sustainability, resilience and logistical point of view. For example, the slaughter of 50,000 reindeer in 12 different abattoirs during 2021 gave rise to 60,000 km of car travel for the control personnel [2]. This is not in line with the United Nations’ Agenda 2030 [3] or the European Green Deal [4], nor with Swedish official sustainability goals [5].

Both AMI and PMI focus on food safety, animal health and animal welfare. Only healthy animals are accepted for slaughter. PMI takes place after the carcass has been degutted and split. The term ‘carcass’ is henceforth used to denote the entire carcass together with accompanying organs.

In Sweden, a specific code system is used for the documentation of findings at PMI, made up of two-digit codes representing the most commonly occurring and important symptoms or findings [6]. Carcasses are inspected by an OA, and any carcasses with suspicious findings are separated from the main line and additional examination is carried out by an OV before a final decision is reached.

With the expansion of 4G and 5G mobile infrastructure, and even fibre-based internet connection, together with advances in video encoding and transmission, PMI at remote sites via video link could become a sustainable alternative to current routines of on-site personnel. This would reduce travel by allowing control personnel to perform inspections at small-scale establishments from a centrally located office. Live video applications have been researched and implemented in human medicine and are currently used in e.g. internet-based medical consultations [7] and various surgical procedures [8, 9]. The technique is also used in clinical veterinary medicine [10], with several developers active in the market.

We have previously shown promising results when evaluating remote PMI, comparing it to on-site PMI, but observed some differences between the two methods [11]. As two different OVs were used in the evaluations (one using each method), any inter-method differences might have been masked by systematic differences in assessments between the OVs. Ideally, the same veterinarian would perform both the on-site and remote inspection of the same carcasses, with sufficient separation in time so as not to remember the initial inspection. However, this is hardly possible when assessing carcasses during on-going slaughter. In addition, most studies on meat inspection would have to be carried out in a real-life setting at a commercial slaughter plant, placing further constraints on what could be achieved in practice. If the same OV could perform PMI both on site and via video transmission with sufficient time in between, this potential problem of difference between OVs could be circumvented. In our previous study we recorded all remote PMI [11], and through the use of these video files, an OV could perform PMI via video on the same carcasses as those previously inspected on site, after a couple of months’ time to reduce recognition memory. By comparing the results of these video inspections to prior results from on-site PMI, the problem of using multiple OVs could be largely mitigated.

The aim of this study was to assess the inter-method, intra-rater reliability between remote and on-site PMI when both are performed by the same OV, by comparing results of inspections performed on video-records of remote, live-video PMI to previous records from PMI performed on site, separately for two OVs.

Methods

Data collection

This study is based in part on previously produced material [11]. Two veterinarians (OVA and OVB), each with several years’ experience of working as OVs, inspected 400 carcasses arrested by OAs for further inspection, using on-site inspection and performing the inspections remotely with the aid of a technician on site. The technician presented the carcass through video, performed any manual tasks the OV deemed necessary, and relayed any requested information back to the OV. Randomly selected falsely detained carcasses (no findings) formed a negative control group of 220 carcasses. The OVs switched methods during the study, inspecting 200 carcasses with each method.

The remote inspections were conducted using video-call software supplemented with augmented reality (Remote Guidance, XMReality AB, Linköping, Sweden), which visualised the carcasses. Each remote PMI was recorded, including the video feed, augmented-reality overlays (when applicable), and all audio communication between the OV and the on-site technician. The videos were stored as h.264 encoded video files with a resolution of 720 × 1280p30, at 3 Mbit/s, a bitrate hard-capped by the XMReality software site [11]. Each video file contained the PMI of a single carcass, totalling 400 videos.

The video recordings were reviewed by OVA and OVB, with each of them reviewing 200 videos produced by the other, i.e., OVA assessed the videos produced by OVB, and vice versa. A schematic overview of the inspections is detailed in Fig. 1.

Fig. 1

Schematic representation of comparisons of on-site to recorded video material of post-mortem meat inspection performed by two official veterinarians (OVs).

Three to six months had passed between the on-site inspections and the review of the videos, to reduce recognition memory. Average video length was 6 min 35 s for the OVB videos viewed by OVA, and 4 min 45 s for the OVA videos viewed by OVB. As a side-note, average time for on-site PMI of the same carcasses was 1 min 53 s [11]. PMI findings were recorded using a modified version of the instruction issued by the Swedish Food Agency [6], to which code 56 for ‘kidney lesion’ (which is normally not recorded in pigs) and code 999 to denote ‘no findings’ were added (Table 1).

Table 1 Codes for documentation of findings in post-mortem inspections of pig carcasses

For each inspection, all codes were stored as binary variables (present or not). The codes represented common lesions or conditions, along with two classifications made by the OV, i.e. perceived false arrest (FA) and total condemnation (TC), where FA indicated that the inspector considered the carcass to have been falsely arrested by the OA and TC that the OV considered the carcass to be unfit for human consumption. The FA and TC classifications were based on the findings at PMI and were mutually exclusive, i.e., a carcass perceived as FA could not be TC, and vice versa. The OA were instructed that only carcasses without any findings be arrested falsely. The OVs also recorded their perceived certainty about the inspection result on a Likert scale from 1 (not at all confident) to 5 (completely confident). The findings and certainty obtained for the video inspections were then compared with data for on-site inspections of the same 200 carcasses by the two OVs from our previous study [11].

Statistical analysis

For both OVs and for each finding or classification, prevalence, observed percentage agreement (joint probability of agreement), Cohen’s kappa [12], Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) [13] and indices of prevalence and bias were calculated, as per recommendations [14] along with 95% confidence intervals. Since no single measure was believed to be an exhaustive representation of agreement, the values were used together to assess the degree of agreement between on-site and video inspection for both OVs. Kappa-based agreement is measured on a scale of 0–1, with a suggested 5-step interpretation, with 0.01–0.20 representing “None to slight agreement”, 0.21–0.40 “Fair agreement”, 0.41–0.60 “Moderate agreement”, 0.61–0.80 “Substantial agreement” and 0.81–1.00 “Almost perfect agreement” [15].

Prevalence was calculated as the average number of reports of a certain finding in both on-site and video inspections, divided by the number of inspected carcasses (n = 200). Calculations on TC were based on the subset of carcasses assessed as not FA during both inspections (n = 79 for OVA, n = 83 for OVB).

All statistical calculations and analyses were performed in R [16]. Cohen’s kappa and PABAK were calculated using the function epi.kappa() in the package epiR [17].

Results

The two OVs scored PABAK values above 0.8 in 21 (OVA) and 19 (OVB) out of the 22 evaluated findings. When comparing PABAK scores between the OVs finding for finding, OVA produced better overall agreement than OVB. For codes 58 (tail lesion) and 84 (parasitic liver lesions), however, OVB scored 0.08 and 0.05 higher, respectively, than OVA. Both OVs produced PABAK values lower than 0.8 for code 999 (no finding; 0.77 and 0.75 for OVA and OVB, respectively). In addition, OVB scored 0.75 for code 56 (kidney lesion) and 0.44 for TC (total condemnation). For OVA, there was a slight increase in average certainty, from 4.32 to 4.51, when comparing on-site to video inspection, while OVB instead showed a marked decrease, from 4.61 to 3.17. For both OVs there was a substantial difference between registrations of FA (falsely arrested) and code 999, with roughly 75% of FA carcasses also bearing code 999.

Detailed results for OVA and OVB are presented in Tables 2 and 3.

Table 2 Intra-rater reliability measurements per finding for OVA.

Table 3 Intra-rater reliability measurements per finding for OVB.

Discussion

When comparing on-site to remote PMI within OVs, both of the two showed higher levels of agreement for almost all findings and classifications than previously observed in comparison of inspection methods across OVs [11]. In all but one code for OVA (code 36, PSE) and five findings for OVB (codes 34, abnormal appearance; 48, emaciation; 88, other liver lesion; FA and TC), PABAK values were higher than in our previous study. Five of the six PABAK values for OVB were only marginally lower than previously [11] and the values were well above 0.8 in both studies. Small differences such as these have very little impact when using kappa-based statistics, due to the rather large steps on the interpretation scale used for kappa [15]. The only substantial difference between this study and previous results was the PABAK value for TC found for OVB, which was 0.44 in this study, compared to 0.50 [11]. The same was true for Cohen’s kappa and percentage agreement, with a large majority of the findings showing higher agreement in this study than previously obtained [11]. We have previously argued that the results of a switch from on-site to remote inspections would have less effect than switching between two OVs on-site [11], and this is underpinned by the fact that agreement in general was even higher in the present study.

Some resulting values for Cohen’s kappa were either 1 or 0. These were considered artefacts, most likely deriving from the extremely low prevalence of the relevant findings (in most cases less than 1%). Due to the overall good health of pigs at slaughter in Sweden it is difficult to produce a sample with high prevalences of all findings. In this study the sample consisted of all carcasses arrested for in-depth inspection, and the prevalences of some findings were still very low. In order to acquire a meaningful absolute number of carcasses with these rare findings the size of the experiment would increase many times over. A sample consisting of pre-selected carcasses would have to be constructed, which would be unfeasible purely because of decomposition of the material over time; by the time you had a large enough sample of rare findings, the first would have degraded. If some findings are rare, mathematically the impact on consumers of poor agreement between methods would be low, since there are so few carcasses that would be affected in absolute numbers. Additionally, it has been suggested that very few findings at PMI are considered hazardous to consumers [18].

In our previous study we made an initial assumption that the two OVs were equally skilled at the start of the study, which was in part contradicted by the results [11]. The present study supports that OVA and OVB cannot be considered completely equal in terms of PMI performance, and substantiates the claim [11] that previous differences between on-site and remote PMI are, at least partly, attributable to the individual OV. Although data collection was standardised, the OVs may still have differed slightly in their inspection routines in time spent, details focused on and manner of decision making. Thus, performing PMI according to someone else’s routine might open up for poor performance. For example, if OVB was very thorough and took the time to inspect in detail, while OVA was quicker to draw a conclusion, when the inspections were viewed by the other OV, OVA would benefit from the extra thoroughness, whereas OVB would be restricted by the shorter videos produced by OVA. This could explain why, when the finding TC scored 0.50 comparing on-site to remote PMI [11], in this study one OV scored 0.45 and the other 0.82, a very marked difference. Had OVB inspected videos of their own on-site inspections the scored would likely have been much higher. The impact of remote PMI, and any shortcomings in agreement between it and on-site PMI has been thoroughly discussed previously [11]. To further expand on this, the results for OVA in the present study shows that remote PMI can actually display “almost perfect agreement” in terms of PABAK across all but one finding (with the last one being close) evaluated both here and previously. This fact alone additionally strengthens the hypothesis that on-site and remote PMI can be interchangeable, assuming a thorough and systematic inspection routine.

It has been noted that longer, more thorough video inspections lead to higher accuracy in human video diagnostics [19]. The pre-recorded PMI videos produced by OVB were on average almost 2 min longer than those produced by OVA, and OVA was more certain when reviewing the videos than during on-site inspections, while OVB was substantially less certain of the video assessments of OVA’s videos. In fact, if the videos that OVB reviewed were too short or otherwise not sufficiently thorough compared with on-site inspection of the same carcasses, this could have contributed to the relatively poor agreement of TC classifications for OVB. If remote PMI was performed in real time, where the OVs could directly affect the inspection routine, the results would likely improve, at least for OVB, who was probably at a disadvantage when reviewing the shorter inspection videos produced by OVA, as discussed above. The differences in inter-method reliability observed between the two OVs highlight the importance of a standardised, thorough inspection routine for remote inspections.

Most findings had similar (but not identical) estimated prevalences for the two OVs. Kappa-based statistics are sensitive to the prevalence of findings, which could have contributed to the observed differences in agreement between the OVs. It cannot be assumed that the distribution of findings was the same for the OVs, since they inspected different carcasses, but it should be similar enough for these differences to be small.

In this study, PABAK showed very good agreement between the methods for both OVs, with only four values below 0.8. However, the relevance of agreement with low prevalence of findings can be questioned, even when using PABAK, since most agreement would stem from negative cases. In our previous results we found that the agreement between OVs tends to be lower for findings that are more subjective, rather than objective assessments [11]. This subjectivity could be said to apply to all findings with PABAK below 0.8 in this study as well.

Another explanation for the lack of perfect agreement between the methods could be that there is some variation, or a random element, in how a person classifies the same finding at repeated inspections, i.e. test-retest agreement. It is not unreasonable to expect a certain degree of variation since no inspector can be assumed to perform completely consistently. The importance of noise; variation in daily individual variation in decision making has been pointed out [20], and differences in experience and knowledge, and in opinion, motivation and dedication, may explain differing agreement between meat inspectors [21]. Motivation and dedication can most likely also vary within an individual, which could cause variations in agreement in this type of comparison. A suitable follow-up study would be test-retest evaluation of the material and the OVs, in order to determine the magnitude of this variation. The discrepancies between registrations of FA and code 999 (which would ideally have been the same) could perhaps also be attributed to these explanations [20, 21]; it is possible the OVs were overly motivated and thorough, being part of a research project, and the OAs were simply performing their normal day-to-day tasks as usual.

This study was primarily based on kappa statistics [12]. Even if the suggested classification of kappa from 0.01 to 0.20 (“none to slight agreement”) to 0.81-1.00 (“almost perfect agreement”) [15] has been criticised as slightly rough and arbitrary [22] it is still the accepted standard. Cohen’s kappa is primarily sensitive to varying prevalence, with low prevalence tending to lower the kappa values, although percentage agreement remains unchanged [13, 23,24,25,26]. It has been pointed out that kappa always assumes a fixed prior probability of rating, either positive or negative [23], and that it always assumes total randomness in the chance agreement [27]. It is likely that most people who guess would at least attempt to make an educated guess rather than “flip a coin”, and the assumption of total randomness is therefore not correct. Instead it can be reasoned that the “true agreement” is probably somewhere between Cohen’s kappa and the observed percentage agreement [27]. We previously concluded that PABAK statistic seems to fit neatly with this criterion [11], and it was therefore used in the present study as well. However, due to the core differences between different agreement measures, we considered it important to report all three values (Cohen’s kappa, PABAK and percentage agreement), in order to give a nuanced picture of the agreement.

Conclusions

This study supports previous findings that the kappa-based agreement between post-mortem inspections of pig carcasses on-site during slaughter and remotely afterwards using video recordings is “almost perfect”. The study also indicates that remote inspections show better agreement when comparisons are made with the same official veterinarian performing both inspections.