逻辑回归自定义正则化：基于权重比例先验的技术咨询

Great question—you’re already thinking in the right direction by connecting regularization to Bayesian priors, which makes customizing it much more intuitive. Let’s break down how to implement this for logistic regression step by step.

1. Translate Domain Knowledge into a Regularization Term #

The core idea is that any Bayesian prior can be converted into a penalty term for your loss function. For MAP estimation, we minimize:
$$\text{Loss} = -\log P(Y|W) + \lambda \cdot R(W)$$
Where:

• $-\log P(Y|W)$ is the standard negative log-likelihood of logistic regression (binary cross-entropy)
• $R(W)$ is the penalty term, derived from the negative log of your custom prior distribution (we can ignore constants that don’t affect optimization)

For weight ratio constraints:

• If you want $W_i = k \cdot W_j$ (e.g., $W_1 = 2W_2$), the penalty term becomes $(W_i - k \cdot W_j)^2$—this penalizes deviations from your desired ratio.
• For multiple linked ratios (e.g., $W_1:W_2=2:1$ and $W_2:W_3=1:0.5$), sum the squared deviations for each pair:
  $$R(W) = (W_1 - 2W_2)^2 + (W_2 - 2W_3)^2$$
• For custom Gaussian priors (e.g., tighter constraints on some weights than others), use a weighted L2 term:
  $$R(W) = \frac{W_1^2}{2\sigma_12} + \frac{W_2^2}{2\sigma_22} + ...$$
  Here, smaller $\sigma$ values mean stricter priors for those weights, aligning with domain confidence.

2. Implementation Options #

You have two flexible paths depending on your tooling preference:

Option 1: Use a Deep Learning Framework (Quick & Flexible) #

Frameworks like PyTorch let you define custom loss functions with minimal boilerplate. Here’s an example enforcing weight ratios:

import torch
import torch.nn as nn
import torch.optim as optim

# Simple logistic regression model
class RatioConstrainedLogReg(nn.Module):
    def __init__(self, num_features):
        super().__init__()
        self.linear = nn.Linear(num_features, 1)
    
    def forward(self, x):
        return torch.sigmoid(self.linear(x))

# Custom loss with ratio penalty
def ratio_loss(y_pred, y_true, model, lambda_reg, target_ratios):
    # Standard binary cross-entropy
    bce_loss = nn.BCELoss()(y_pred, y_true)
    
    # Extract weights (exclude bias for ratio constraints)
    weights = model.linear.weight.squeeze()
    
    # Build penalty from target ratios (example: W0=2*W1, W1=1.5*W2)
    penalty = 0.0
    penalty += (weights[0] - target_ratios[0] * weights[1])**2
    penalty += (weights[1] - target_ratios[1] * weights[2])**2
    
    return bce_loss + lambda_reg * penalty

# Example usage
X = torch.randn(100, 3)  # 100 samples, 3 features
y = torch.randint(0, 2, (100, 1)).float()

model = RatioConstrainedLogReg(3)
optimizer = optim.Adam(model.parameters(), lr=0.001)
lambda_reg = 0.1  # Balance data fit and prior constraints
target_ratios = [2.0, 1.5]  # W0:W1=2, W1:W2=1.5

# Training loop
for epoch in range(1000):
    optimizer.zero_grad()
    y_pred = model(X)
    loss = ratio_loss(y_pred, y, model, lambda_reg, target_ratios)
    loss.backward()
    optimizer.step()
    
    if epoch % 100 == 0:
        print(f"Epoch {epoch} | Loss: {loss.item():.4f}")

# Check final weight ratios
final_weights = model.linear.weight.squeeze().detach().numpy()
print(f"\nFinal weights: {final_weights.round(2)}")
print(f"Ratio W0/W1: {final_weights[0]/final_weights[1]:.2f} (target: 2.0)")
print(f"Ratio W1/W2: {final_weights[1]/final_weights[2]:.2f} (target: 1.5)")

Option 2: Manual Gradient Descent (Full Control) #

If you prefer to avoid frameworks, implement gradient descent manually to calculate gradients for both the standard loss and your custom penalty:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def custom_log_reg(X, y, lambda_reg, target_ratios, epochs=1000, lr=0.01):
    n_samples, n_features = X.shape
    weights = np.zeros(n_features)
    bias = 0
    
    for _ in range(epochs):
        # Forward pass
        z = np.dot(X, weights) + bias
        y_pred = sigmoid(z)
        
        # Gradient of cross-entropy loss
        dw = (1/n_samples) * np.dot(X.T, (y_pred - y))
        db = (1/n_samples) * np.sum(y_pred - y)
        
        # Gradient of ratio penalty term
        dw_penalty = np.zeros_like(weights)
        # Derivatives for W0=2*W1 and W1=1.5*W2
        dw_penalty[0] = 2 * (weights[0] - target_ratios[0] * weights[1])
        dw_penalty[1] = 2*(weights[1]-target_ratios[1]*weights[2]) - 2*target_ratios[0]*(weights[0]-target_ratios[0]*weights[1])
        dw_penalty[2] = -2 * target_ratios[1] * (weights[1] - target_ratios[1] * weights[2])
        
        # Update weights and bias
        weights -= lr * (dw + lambda_reg * dw_penalty)
        bias -= lr * db
    
    return weights, bias

# Example usage
X = np.random.randn(100, 3)
y = np.random.randint(0, 2, 100)

weights, bias = custom_log_reg(X, y, lambda_reg=0.1, target_ratios=[2.0, 1.5])
print(f"Final weights: {weights.round(2)}")

3. Key Tips #

• Tune $\lambda$: Start with small values and increase until weight ratios align with your domain knowledge without hurting validation performance.
• Normalize Features: Standardize features (mean=0, std=1) first—different scales make ratio constraints hard to interpret and enforce.
• Validate: Always check that the resulting weights make sense for your use case, and test model performance on unseen data to avoid over-constraining.

内容的提问来源于stack exchange，提问作者claudius