Hanna

Sentinel lymph node biopsy (SLNB) is a procedure performed for melanoma patients to check whether their melanoma has spread beyond their original tumor site. This information is used for determining an appropriate treatment plan and predicting long term outcomes for patients. However, there are subsets of patients who do not undergo SLNB for a variety reasons: the tumor is considered thin, the patient is elderly, the melanoma is already late stage, or the patient simply elects to not have the procedure.

Patients in the dataset are characterized by having: age between 20 and 89 years at time of diagnosis, melanomas greater than or equal to 0.45mm, diagnosis of first and only melanoma in 2010 or 2011, and are a part of a cancer registry. A subset of patients were excluded due to coding inconsistencies. Variables available for modeling include sex, age (coded by decade), melanoma thickness (mm), primary site, histology, ulcer status, and mitotic rate (present or absent).

This analysis aims to predict 5 year melanoma-specific survival, as a binary outcome, for patients who did not undergo SLNB. While SLNB provides important information that informs long-term prognosis, it’s important to maintain a high predictive ability in the absence of this variable. Here we leverage other risk factors to investigate how well they perform alone, and jointly. I use both a logistic regression model, and a machine learning Naive Bayes model to generate predictions, and compare the model performance by AUC and model calibration. The models are internally validated using a bootstrap procedure.

In the sample of 3331 subjects, 232 (7%) died due to their melanoma within five years of diagnosis. Table 1 looks at the characteristics comparing those who survived more than 5 years and those who did not for categorical variables sex, mitotic rate status, primary site, histology and ulcer status. A chi-square test of independence is used for each variable to test whether the distributions are comparable between the two groups (i.e. whether each variable is independent from survival status or there is an association between the two) . Rejecting the null indicates a significant association between the variable and survival status.

Table 1 indicates that a strong majority of subjects who die within 5 years of their melanoma are male and had some degree of melanoma cell mitosis present (mitosis is a marker of how fast the cells are dividing). The most frequent type of histological classifications were malignant melanoma and nodular melanoma among those who died. About half had presence of an ulcer compared to just 6% in the group that survived beyond 5 years. Primary site 3 was most frequent among both groups (variable codes unknown).

Table 2 compares the means of the continuous variables age at diagnosis (coded by decade) and thickness in mm between those who survived and those who did not, along with t-tests for a difference in means. Those who die within 5 years from their melanoma are significantly older (by more than 10 years, on average) at the time of diagnosis, and had melanomas, on average, two millimeters greater - again a statistically significant difference. It is also interesting to note that the standard deviations for age are comparable between the two groups, but the standard deviation for thickness is over three times greater in the group that died within 5 years.

With a better understanding of the study population, we conduct a univariate analysis to understand the impact of each variable alone as a predictor before undertaking the full multivariate analysis. Letting $Y=1$ denote death from melanoma within 5 years and $Y=0$ denote alive at 5 years, each variable, $X$, enters a logistic regression model with $Y$ as the outcome: $\log\Bigg[\frac{P(Y=1|X)}{P(Y=0|X)}\Bigg]=\beta_0+\beta_1X$ The quantity $e^{\beta_1}$ represents the multiplicative increase (or decrease) in the odds of death for a one unit increase in $X$. For categorical $X$, each category has its own regression coefficient except for the chosen referent category for identifiability purposes. Table 3 contains the estimated coefficients, standard errors, and Wald tests to assess the significance of each predictor for the seven model fits. All variables appear to be significant on their own; having mitosis present, a nodular melanoma, a acral lentiginous melanoma, or an ulcer present stand out as increasing the odds of death by more than 5 times that of the respective referent groups.

Fitting the logistic regression model allows us to construct an ROC curve and corresponding AUC for each predictor. This is important to consider separately because significant association, as seen in Table 3, will not necessarily imply high predictive ability. Given the model fits above, we can estimate each subjects probability of death as $\hat{P}(Y=1|X)$ for each variable. Using this quantity as a ‘diagnostic’, we calculate the operating characteristics sensitivity and specificity for all possible values ($c$) of $\hat{P}\in [0,1]$. $Sensitivity=Pr(\hat{P}\geq c|Y=1)$ $Specificity=Pr(\hat{P}< c|Y=0)$ Sensitivity represents the true positive fraction, and specificity the true negative fraction, and tell us for a given cut point $c$, how well the predictor variable from the logistic regression model discriminates between those who die within 5 years and those who do not.

A plot of 1-specificity versus sensitivity for all $c$ gives rise to the ROC curve. The area under the ROC curve - the AUC - is an overall statistic, that we use to inform how well the model predicts death. A value of AUC=1 means 100% of the outcomes were correctly predicted by the fitted model. If the ROC curve follows the 45 degree line, the AUC will be close to 0.5, indicating no predictive ability of the model - effectively a coin toss. Figure 1 below shows the ROC curves for each predictor variable, along with their AUC. Thickness (of the melanoma) has the largest AUC at 0.87 indicating it best predicts death within 5 years, despite the fact that its effect size in Table 3 was not the largest. The AUC can also be interpreted as the probability that a randomly chosen subject with $Y=1$ has a higher $\hat{P}$ value than a randomly chosen subject with $Y=0$. The AUC resulting from the training data (here, the entire dataset) will inherently be overly ‘optimistic’, so it is important that we use a validation procedure later when comparing predictive performance between the two models.

Understanding the strength of these predictors alone, we now ask how strong they are when used in conjunction. We will consider two different prediction models: the logistic regression model again, and the naive Bayes model. The latter is considered a machine learning approach, and is more involved than the logistic regression model. One question we are interested in answering is whether our ability to estimate model tuning parameters leads to any substantial payoff in terms of our ability to predict death within 5 years.

We fit the multivariate regression model:
$\log\Bigg[\frac{P(Y=1|X)}{P(Y=0|X)}\Bigg]=\beta_0+\beta_1*Sex+ \beta_2*Age+\beta_3*Ulcer+\beta_4*Mitosis + \beta_5*Thickness +$ $\vec{\boldsymbol{\beta}_6}*PSite+\vec{\boldsymbol{\beta}_7}*Histology$ where $\boldsymbol{\vec{\beta}_6}=[\beta_{6,1},\beta_{6,2},\beta_{6,3},\beta_{6,4}]$ represents the 5 primary sites (P1 is the referent group), and $\boldsymbol{\vec{\beta}_7}=[\beta_{7,1},...,\beta_{7,9}]$ represents the 9 histologies (malignant melanoma is the referent group).

When taken together, sex is no longer significant in the model. Age, ulcer status, mitosis status, and thickness are all highly significant, while only the P2 primary site is significant relative to P1. Nodular Melanoma and Epitheliod Cell Melanoma are significant relative to the Malignant Melanoma histology. Again, significant association does not imply high predictive ability so we evaluate the predictive ability by looking at the AUC. Under the multivariate model, the AUC increased to 0.90, though this will be corrected through internal bootstrap validation. Bootstrap validation is selected instead of splitting the dataset into a training and test set because of the low event rate. The logistic regression and Naive Bayes model will be validated concurrently.

It is important to examine model calibration prior to validation, which looks at whether the model is predicting an accurate number of events for intervals of the risk score, $\hat{P}$. For example, we’d like to see that subjects with a risk score between 0 and 0.1 contribute about 12 out of 232 (0.05%) of the total number of observed events, since 0.05 is the midpoint of the risk score interval. We examine calibration using a calibration plot; a plot that follows the 45-degree line indicates good model calibration. A model that predicts well but is poorly calibrated is ultimately not useful, as it is over-predicting the outcome of interest for some individuals and under-predicting for others. The logistic regression model has good model calibration, noted by the blue line that roughly follows the 45-degree line in Figure 3. The exception is the high risk groups, where there is some over-prediction and under-prediction.

The logistic regression model is made up of a linear combination of variables, while the Naive Bayes model is represented entirely by probability statements: $P(Y|\mathbf{X})=\frac{P(\mathbf{X}|Y)\times P(Y)}{P(\mathbf{X})}$ When $X$ is categorical, $P(X)$ and $P(X|Y)$ is estimated using table proportions, and the prior class probability $P(Y)$ is also estimated from the data. For continuous $X$, the distribution can either be estimated using a normal kernel, or a nonparametric kernel; this choice serves as the tuning parameter for the model. Additionally, we can include a tuning parameter for smoothing any marginal probabilities that are equal to 0, called the Laplace correction which adds a small value to the probabilities. We will consider values 0 (no smoothing), 0.5, 1, and 2. One limitation of this model is that in order to improve computation, we assume the covariates are independent, which is unlikely to be true. However it enables us to calculate joint probabilities as the product of marginal probabilities; studies have shown the Naive Bayes model still maintains good model performance relative to other machine learning methods under the independence assumption.

We use leave-one-out cross validation to determine the tuning parameters using the AUC as the performance metric. The model performed the same regardless of Laplace correction, so we select 0, no smoothing. Model performance was better using kernel density estimates for the covariate distributions. The AUC to evaluate the predictive ability of the Naive Bayes model is 0.90, the same as that of the logistic regression model.

Looking at the calibration plot, however, reveals that the model is poorly calibrated. There is vast overestimation for a majority of the risk intervals. Fitting the another Naive Bayes model to the original model’s predictions is a tool for recalibrating a model. Doing so overcorrected and resulted in excessive underestimation (data not shown), so we retain the original Naive Bayes model.

Now that both models have been developed, we use internal bootstrap validation to correct the model performance, since it will always be overly optimistic when using the exact data the model was built on.

After validation, the predictive ability of the logistic regression and naive Bayes model reduced to 0.88 and 0.89. Given the relatively small dataset, and the straightforwardness of the problem, I saw no reason a priori that the machine learning model would outperform the logistic regression model. These results confirm this, as the models perform almost equally in terms of predictive performance. The logistic regression model earns a leg up though, in having much better model calibration. It is well known that Naive Bayes models, in general, do not tend to have great model calibration.

In the absense of a sentinel lymph node biopsy and knowing whether the cancer has entered the lymphatic system, we still have a good ability to predict death within 5 years. We can infer from the univariate analysis that thickness is the variable that contributes the most to the predictive ability, which makes sense given that thicker lesions would correspond to more advanced and aggressive disease. We were able to confirm that we can maintain a high predictive ability in the absense of information from a SLNB, which is important given there are multiple subsets of patients who will not have this information. One limitation of the presented work, and a option to extend the analysis, is to look at prediction on an individual level. The AUC is a population-based metric and tells us here that we are predicting almost the right number of deaths across the study population. It does not tell us whether those predicted to die within 5 years are those who actually died within 5 years in the dataset. A confusion matrix would help to understand this aspect of the predictions.