Objectives: To investigate uniformity and pitfalls in structured radiological staging of rectal cancer.

Method: Twenty-one radiologists (12 countries) staged 75 rectal cancers on MRI using a structured reporting template. Interobserver agreement (IOA) was calculated as the percentage agreement between readers (categorical variables) and Krippendorff’s α (continuous variables). Agreement with an expert consensus served as a surrogate standard of reference to estimate diagnostic accuracy. Polychoric correlation coefficients were used to assess correlations between diagnostic confidence and accuracy (=agreement with expert consensus).

Results: Uniformity to diagnose high-risk (≥cT3 ab) versus low-risk (≤cT3 cd) cT-stage, cN0 versus cN+, lateral nodes and tumour deposits, MRF and sphincter involvement, and solid versus mucinous tumours was high with IOA > 80% in the majority of cases (and >80% agreement with expert consensus). Results for assessing extramural vascular invasion, cT-stage (cT1-2/cT3/cT4a/cT4b), cN-stage (cN0/N1/N2), relation to the peritoneal reflection, extent of sphincter involvement (internal/intersphincteric/external) and morphology (solid/annular/semi-annular) were considerably poorer. IOA was high (α = 0.72–0.84) for tumour height/length and extramural invasion depth, but low for tumour-MRF distance and number of (suspicious) nodes (α = 0.05–0.55). There was a significant positive correlation between diagnostic confidence and accuracy (=agreement with expert consensus) (p < 0.001-p = 0.003).

Conclusions:– Several staging items lacked sufficient
reproducibility.
– Results for cT- and N-staging g improved when using a dichotomized stratification.
– Considering the significant correlation between diagnostic confidence and accuracy, a confidence level may be incorporated into structured reporting for specific items with low reproducibility.

Advances in knowledge: Although structured reporting aims to achieve uniformity in reporting, several items lack sufficient reproducibility and might benefit from dichotomized assessment and incorporating confidence levels.