技术求助：不借助Scikit-Learn从零构建Python版Support Vector Machine

Awesome choice—building an SVM from scratch is one of the best ways to really grok how it works, instead of just calling a library function. Let's walk through this step by step, starting with core concepts, then moving to runnable code you can tweak and experiment with.

Core SVM Concepts to Understand First #

Before diving into code, let's recap the basics that drive the implementation:

• Hyperplane: For a dataset with n features, this is a line/plane defined by w·x + b = 0, where w is the weight vector and b is the bias term. Our goal is to find the hyperplane that separates two classes with the widest possible margin.
• Hinge Loss: The loss function that penalizes misclassifications and encourages a wide margin. It’s defined as max(0, 1 - y*(w·x + b)), where y is the true label (we use -1 and 1 for classes, not 0 and 1 here).
• Regularization: To prevent overfitting, we add an L2 regularization term (1/2)*||w||² to the loss. The total cost is the average hinge loss plus this regularization term, scaled by a parameter (inversely related to scikit-learn’s C).

Step 1: Implement the SVM Class #

We’ll create a class with fit (training) and predict (inference) methods, using stochastic gradient descent to optimize our parameters.

import numpy as np

class CustomLinearSVM:
    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iters=1000):
        self.lr = learning_rate
        self.lambda_param = lambda_param  # Equivalent to 1/C in scikit-learn
        self.n_iters = n_iters
        self.w = None
        self.b = None

    def fit(self, X, y):
        n_samples, n_features = X.shape
        
        # Convert labels to -1 and 1 (required for hinge loss)
        y_ = np.where(y <= 0, -1, 1)
        
        # Initialize weights and bias
        self.w = np.zeros(n_features)
        self.b = 0
        
        # Stochastic Gradient Descent loop
        for _ in range(self.n_iters):
            for idx, x_i in enumerate(X):
                # Check if the sample is correctly classified within the margin
                margin_condition = y_[idx] * (np.dot(x_i, self.w) + self.b) >= 1
                
                if margin_condition:
                    # Only update weights for regularization
                    self.w -= self.lr * (self.lambda_param * self.w)
                else:
                    # Update both weights and bias to correct the misclassification
                    self.w -= self.lr * (self.lambda_param * self.w - np.dot(x_i, y_[idx]))
                    self.b -= self.lr * (-y_[idx])

    def predict(self, X):
        # Compute linear output and return class sign (-1 or 1)
        linear_output = np.dot(X, self.w) + self.b
        return np.sign(linear_output)

Step 2: Test the Model #

Let’s validate our implementation with a simple 2D dataset (we’ll use scikit-learn just to generate test data, not for the SVM itself):

from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Generate a binary classification dataset
X, y = make_blobs(n_samples=100, n_features=2, centers=2, cluster_std=1.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train our custom SVM
svm = CustomLinearSVM(learning_rate=0.001, lambda_param=0.01, n_iters=1000)
svm.fit(X_train, y_train)

# Predict and evaluate accuracy
predictions = svm.predict(X_test)
# Convert predictions back to 0/1 if needed (since our model returns -1/1)
predictions = np.where(predictions == -1, 0, 1)
print(f"Custom SVM Accuracy: {accuracy_score(y_test, predictions):.2f}")

Key Notes for Improvement #

• Linear Only: This is a linear SVM. To handle non-linear data, you’d need to add kernel functions (like RBF or polynomial) that transform the data into a higher-dimensional space where it’s linearly separable.
• Vectorization: The current fit method loops over individual samples. For faster training, you could vectorize the operations to process all samples at once (batch gradient descent) or in mini-batches.
• Feature Scaling: SVMs are sensitive to feature scales—normalizing/standardizing your data before training will improve performance.
• Hyperparameter Tuning: Adjust learning_rate, lambda_param, and n_iters to get better results for your specific dataset.

Next Steps to Explore #

1. Implement batch/mini-batch gradient descent instead of stochastic.
2. Add kernel functions (e.g., RBF) to support non-linear classification.
3. Optimize the training loop with early stopping to prevent overfitting.

I hope this gives you a clear understanding of how SVMs work under the hood. Feel free to ask if you want to dive deeper into any specific part!

内容的提问来源于stack exchange，提问作者Chandan Shaw