Conventional Strategy

Many current implementations on survival evaluation begin off with a dataset containing one commentary per particular person (sufferers in a well being research, staff within the attrition case, purchasers within the shopper churn case, and so forth). For these people we usually have two key variables: one signaling the occasion of curiosity (an worker quitting) and one other measuring time (how lengthy they’ve been with the corporate, as much as both at this time or their departure). Along with these two variables, we then have explanatory variables with which we goal to foretell the danger of every particular person. These options can embody the job function, age or compensation of the worker, for instance.

Shifting on, most implementations on the market take a survival mannequin (from less complicated estimators resembling Kaplan Meier to extra complicated ones like ensemble fashions and even neural networks), match them over a prepare set after which consider over a check set. This train-test break up is often carried out over the person observations, typically making a stratified break up.

In my case, I began with a dataset that adopted a number of staff in an organization month-to-month till December 2023 (in case the worker was nonetheless on the firm), or till the month they left the corporate — the occasion date:

With a purpose to adapt my knowledge to the survival case, I took the final commentary of every worker as proven within the image above (the blue dots for energetic staff, and the pink crosses for workers who left). At that time for every worker, I recorded whether or not the occasion had occurred at that date or not (in the event that they had been energetic or if that they had left), their tenure in months at the moment, and all their explanatory variables. I then carried out a stratified train-test break up over this knowledge, like this:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
# We load our dataset with a number of observations (record_date) per worker (employee_id)
# The occasion column signifies if the worker left on that given month (1) or if the worker was nonetheless energetic (0)
df = pd.read_csv(f'{FILE_NAME}.csv')
# Making a label the place optimistic occasions have tenure and destructive occasions have destructive tenure - required by Random Survival Forest
df_model['label'] = np.the place(df_model['event'], df_model['tenure_in_months'], - df_model['tenure_in_months'])
df_train, df_test = train_test_split(df_model, test_size=0.2, stratify=df_model['event'], random_state=42)

After performing the break up, I proceeded to suit a mannequin. On this case, I selected to experiment with a Random Survival Forest utilizing the scikit-survival library.

from sklearn.preprocessing import OrdinalEncoder
from sksurv.datasets import get_x_y
from sksurv.ensemble import RandomSurvivalForest
cat_features = [] # record of all the explicit options
options = [] # record of all of the options (each categorical and numeric)
# Categorical Encoding
encoder = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)
encoder.match(df_train[cat_features])
df_train[cat_features] = encoder.rework(df_train[cat_features])
df_test[cat_features] = encoder.rework(df_test[cat_features])
# X & y
X_train, y_train = get_x_y(df_train, attr_labels=['event','tenure_in_months'], pos_label=1)
X_test, y_test = get_x_y(df_test, attr_labels=['event','tenure_in_months'], pos_label=1)
# Match the mannequin
estimator = RandomSurvivalForest(random_state=RANDOM_STATE)
estimator.match(X_train[features], y_train)
# Retailer predictions
y_pred = estimator.predict(X_test[features])

After a fast run utilizing the default settings of the mannequin, I used to be thrilled with the check metrics I noticed. To start with, I used to be getting a concordance index above 0.90 within the check set. The concordance index is a measure of how effectively the mannequin predicts the order of occasions: it displays whether or not staff predicted to be at excessive danger had been certainly those leaving the corporate first. An index of 1 corresponds to good prediction accuracy, whereas an index of 0.5 signifies a prediction no higher than random probability.

I used to be significantly fascinated by seeing if the workers who left within the check set matched with probably the most dangerous staff in keeping with the mannequin. Within the case of the Random Survival Forest, the mannequin returns the danger scores of every commentary. I took the proportion of staff who left the corporate within the check set, and used it to filter probably the most dangerous staff in keeping with the mannequin. The outcomes had been very strong, with the workers flagged with probably the most danger matching nearly completely with the precise leavers, with an F1 rating above 0.90 within the minority class.

from lifelines.utils import concordance_index
from sklearn.metrics import classification_report
# Concordance Index
ci_test = concordance_index(df_test['tenure_in_months'], -y_pred, df_test['event'])
print(f'Concordance index:{ci_test:0.5f}n')
# Match probably the most dangerous staff (in keeping with the mannequin) with the workers who left
q_test = 1 - df_test['event'].imply()
thr = np.quantile(y_pred, q_test)
risky_employees = (y_pred >= thr) * 1
print(classification_report(df_test['event'], risky_employees))

Getting +0.9 metrics on the primary run ought to set off an alarm: was the mannequin actually capable of predict whether or not an worker was going to remain or depart with such confidence? Think about this: we submit our predictions saying which staff are most definitely to depart. Nonetheless, a pair months go by, and HR then reaches us nervous, saying that the individuals who left over the last interval, didn’t precisely match with our predictions, no less than on the fee it was anticipated from our check metrics.

We’ve two primary issues right here: the primary one is that our mannequin isn’t extrapolating fairly in addition to we thought. The second, and even worse, is that we weren’t capable of measure this lack of efficiency. First, I’ll present a easy manner we are able to estimate how effectively our mannequin is actually extrapolating, after which I’ll discuss one potential motive it might be failing to take action, and methods to mitigate it.

Estimating Generalization Capabilities

The important thing right here is accessing panel knowledge, that’s, a number of data of our people over time, up till the time of occasion or the time the research ended (the date of our snapshot, within the case of worker attrition). As a substitute of discarding all this data and maintaining solely the final document of every worker, we may use it to create a check set that can higher replicate how the mannequin performs sooner or later. The thought is kind of easy: suppose we’ve got month-to-month data of our staff up till December 2023. We may transfer again, say, 6 months, and fake we took the snapshot in June as an alternative of December. Then, we might take the final commentary for workers who left the corporate earlier than June 2023 as optimistic occasions, and the June 2023 document of staff who survived past that date as destructive occasions, even when we already know a few of them ultimately left afterwards. We’re pretending we don’t know this but.

As the image above reveals, I take a snapshot in June, and all staff who had been energetic at the moment are taken as energetic. The check dataset takes all these energetic staff at June with their explanatory variables as they had been on that date, and takes the most recent tenure they achieved by December:

test_date = '2023-07-01'
# Choosing coaching knowledge from data earlier than the check date and taking the final commentary per worker
df_train = df[df.record_date < test_date].reset_index(drop=True).copy()
df_train = df_train.groupby('employee_id').tail(1).reset_index(drop=True)
df_train['label'] = np.the place(df_train['event'], df_train['tenure_in_months'], - df_train['tenure_in_months'])
# Making ready check knowledge with data of energetic staff on the check date
df_test = df[(df.record_date == test_date) & (df['event']==0)].reset_index(drop=True).copy()
df_test = df_test.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.drop(columns = ['tenure_in_months','event'])
# Fetching the final tenure and occasion standing for workers within the check dataset
df_last_tenure = df[df.employee_id.isin(df_test.employee_id.unique())].reset_index(drop=True).copy()
df_last_tenure = df_last_tenure.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.merge(df_last_tenure[['employee_id','tenure_in_months','event']], how='left')
df_test['label'] = np.the place(df_test['event'], df_test['tenure_in_months'], - df_test['tenure_in_months'])

We match our mannequin once more on this new prepare knowledge, and as soon as we end we make our predictions for all staff who had been energetic on June. We then evaluate these predictions to the precise end result of July — December 2023 — that is our check set. If these staff we marked as having probably the most danger left throughout the semester, and people we marked as having the bottom danger didn’t depart, or left relatively late within the interval, then our mannequin is extrapolating effectively. By shifting our evaluation again in time and leaving the final interval for analysis, we are able to have a greater understanding of how effectively our mannequin is generalizing. After all, we may take this one step additional and carry out some kind of time-series cross validation. For instance, we may iterate this course of many instances, every time shifting 6 months again in time, and evaluating the mannequin’s accuracy over a number of time frames.

After coaching our mannequin as soon as once more, we now see a drastic lower in efficiency. To start with, the concordance index is now round 0.5 — equal to that of a random predictor. Additionally, if we attempt to match the ‘n’ most dangerous staff in keeping with the mannequin with the ‘n’ staff who left within the check set, we see a really poor classification with a 0.15 F1 for the minority class:

So clearly there’s something fallacious, however no less than we are actually capable of detect it as an alternative of being misled. The primary takeaway right here is that our mannequin performs effectively with a conventional break up, however doesn’t extrapolate when doing a time-based break up. This can be a clear signal that a while bias could also be current. In brief, time-dependent data is being leaked and our mannequin is overfitting over it. That is widespread in circumstances like our worker attrition drawback, when the dataset comes from a snapshot taken at some date.

Time Bias

The issue cuts right down to this: all our optimistic observations (staff who left) belong to previous dates, and all our destructive observations (at the moment energetic staff) are all measured on the identical date — at this time. If there’s a single characteristic that reveals this to the mannequin, then as an alternative of predicting danger we will probably be predicting if an worker was recorded in December 2023 or earlier than. This might be very refined. For instance, one characteristic we might be utilizing is the engagement rating of the workers. This characteristic may effectively present some seasonal patterns, and measuring it on the similar time for energetic staff will certainly introduce some bias within the mannequin. Perhaps in December, throughout the vacation season, this engagement rating tends to lower. The mannequin will see a low rating related to all energetic staff, so it might study to foretell that each time the engagement runs low, the churn danger additionally goes down, when in truth it must be the alternative!

By now, a easy but fairly efficient answer for this drawback must be clear: as an alternative of taking the final commentary for every energetic worker, we may simply decide a random month from all their historical past inside the firm. It will strongly scale back the possibilities of the mannequin selecting on any temporal patterns that we don’t need it to overfit on:

Within the image above we are able to see that we are actually spanning a broader set of dates for the energetic staff. As a substitute of utilizing their blue dots at June 2023, we take the random orange dots as an alternative, and document their variables on the time, and the tenure that they had thus far within the firm:

np.random.seed(0)
# Choose coaching knowledge earlier than the check date
df_train = df[df.record_date < test_date].reset_index(drop=True).copy()
# Create an indicator for whether or not an worker ultimately churns inside the prepare set
df_train['indicator'] = df_train.groupby('employee_id').occasion.rework(max)
# Isolate data of staff who left, and retailer their final commentary
churn = df_train[df_train.indicator==1].reset_index(drop=True).copy()
churn = churn.groupby('employee_id').tail(1).reset_index(drop=True)
# For workers who stayed, randomly decide one commentary from their historic data
keep = df_train[df_train.indicator==0].reset_index(drop=True).copy()
keep = keep.groupby('employee_id').apply(lambda x: x.pattern(1)).reset_index(drop=True)
# Mix churn and keep samples into the brand new coaching dataset
df_train = pd.concat([churn,stay], ignore_index=True).copy()
df_train['label'] = np.the place(df_train['event'], df_train['tenure_in_months'], - df_train['tenure_in_months'])
del df_train['indicator']
# Put together the check dataset equally, utilizing solely the snapshot from the check date
df_test = df[(df.record_date == test_date) & (df.event==0)].reset_index(drop=True).copy()
df_test = df_test.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.drop(columns = ['tenure_in_months','event'])
# Get the final recognized tenure and occasion standing for workers within the check set
df_last_tenure = df[df.employee_id.isin(df_test.employee_id.unique())].reset_index(drop=True).copy()
df_last_tenure = df_last_tenure.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.merge(df_last_tenure[['employee_id','tenure_in_months','event']], how='left')
df_test['label'] = np.the place(df_test['event'], df_test['tenure_in_months'], - df_test['tenure_in_months'])

We then prepare our mannequin as soon as once more, and consider it over the identical check set we had earlier than. We now see a concordance index of round 0.80. This isn’t the +0.90 we had earlier, but it surely positively is a step up from the random-chance stage of 0.5. Concerning our curiosity in classifying staff, we’re nonetheless very far off the +0.9 F1 we had earlier than, however we do see a slight enhance in comparison with the earlier strategy, particularly for the minority class.