Hey there! It makes sense that KNN is giving you a high log loss (0.9) here—KNN often struggles with mixed-type features (dates, categorical locations) and doesn't naturally capture the complex relationships between booking behavior and cabin class. Let's dive into models that are better suited for this task, along with actionable tips:

1. Gradient Boosting Trees (XGBoost, LightGBM, CatBoost) #

These are my top picks for your use case—they excel at handling mixed feature types, automatically learn feature interactions (like how booking lead time correlates with cabin choice), and perform well even with smaller datasets like your 1-month sample.

• Key steps:
  • First, level up your feature engineering: calculate booking_lead_days (departure date - booking date), bin departure time into segments (early morning, midday, evening), and flag high-traffic routes (origin-destination pairs with frequent bookings). These features will give the model far more signal to work with.
  • For multi-class log loss, set the objective to multi:softprob in XGBoost/LightGBM (this outputs class probabilities, which log loss relies on). Tune hyperparameters like learning rate (eta), tree depth, and subsample ratio to avoid overfitting.
  • CatBoost is a great choice if you want to skip manual categorical encoding—it handles origin/destination labels automatically while reducing overfitting.

2. Regularized Logistic Regression #

Logistic regression is a workhorse for classification tasks where probability calibration matters (and log loss depends heavily on well-calibrated probabilities). It's lightweight and interpretable, which is a bonus.

• Key steps:
  • Encode categorical features (origin, destination) using target encoding (better than one-hot for small datasets) or one-hot if you have a limited number of unique values. Convert date/time features to numerical values: day of week, hour of departure, booking lead time as a number.
  • Use LogisticRegression with multi_class='multinomial' and a solver like saga (handles large feature spaces well). Add L2 regularization (penalty='l2') to prevent overfitting—critical since your dataset is only 1 month of data.

3. Multi-Layer Perceptron (MLP) #

If you're open to neural networks, an MLP can learn more complex feature interactions once you've properly preprocessed your data. It's especially useful if you plan to scale your dataset later.

• Key steps:
  • Convert all features to numerical form: use embeddings for categorical features (origin/destination) or one-hot encoding, and standardize numerical features (like booking lead time, departure hour).
  • Use MLPClassifier with a small hidden layer setup (e.g., hidden_layer_sizes=(64, 32)) to avoid overfitting. Add dropout layers if you're using a framework like TensorFlow/PyTorch, or use the built-in alpha parameter for L2 regularization in scikit-learn's MLP.

Bonus Tips to Further Reduce Log Loss #

• Probability Calibration: No matter which model you choose, use CalibratedClassifierCV to calibrate your model's probability outputs. Log loss penalizes poorly calibrated probabilities heavily, so this step can directly lower your score.
• Check for Class Imbalance: If one cabin class (e.g., first class) has far fewer samples than others, use class weights (class_weight='balanced' in scikit-learn) or oversample minority classes with SMOTE to give the model better exposure to all classes.
• Feature Selection: Use feature importance scores (from XGBoost/CatBoost) or mutual information to drop irrelevant features—less noise means the model can focus on the signals that actually predict cabin class.

内容的提问来源于stack exchange，提问作者user2905648