无公开WebSocket API时，如何获取网站数据并用于Node/Angular？

Great question—this is a super common scenario when you want to tap into real-time data that a site uses internally but doesn’t expose publicly. Let’s break this down step by step so you can get that data into your Node.js or Angular project.

Step 1: Reverse-Engineer the WebSocket Connection #

First, you need to extract all the details required to mimic the site’s WebSocket connection using Chrome DevTools (which you’re already using to see the frames):

• Grab the connection URL: In the Network > WS tab, click the active connection and check the "Request URL" under Headers—this is the wss:// endpoint you’ll connect to.
• Copy critical request headers: Look at the "Request Headers" section. You’ll almost always need Origin (matches the site’s domain) and Cookie (if the site requires authentication). Some sites also use custom headers like Authorization or X-Session-Token—copy those too.
• Analyze the data format: Switch to the Frames tab to see how data is sent/received. Is it JSON? Binary? A custom string format? Note any initial "subscribe" messages the site sends to start receiving updates—you’ll need to replicate this. Also watch for ping/pong frames (many sites use these to keep connections alive; you’ll need to respond to pings or send your own to avoid being disconnected).

Step 2: Connect & Fetch Data in Node.js #

The easiest way to handle WebSockets in Node.js is with the ws library. Here’s a working example:

1. Install the library:

   npm install ws
   

2. Write the connection code:

   const WebSocket = require('ws');
   
   // Replace with values from DevTools
   const WS_ENDPOINT = 'wss://target-site.com/internal-realtime-endpoint';
   const CONNECTION_HEADERS = {
     'Origin': 'https://target-site.com',
     'Cookie': 'your-auth-cookie-from-devtools',
     // Add any other required headers here
   };
   
   const ws = new WebSocket(WS_ENDPOINT, { headers: CONNECTION_HEADERS });
   
   ws.on('open', () => {
     console.log('Connected to target WebSocket!');
     // Send the same subscribe message you saw in DevTools Frames
     ws.send(JSON.stringify({ action: 'subscribe', channel: 'live-data-feed' }));
   });
   
   ws.on('message', (data) => {
     // Process the incoming data (parse JSON, transform, etc.)
     const parsedData = JSON.parse(data.toString());
     console.log('Received real-time update:', parsedData);
     // Pass this data to your Node.js service, database, or frontend
   });
   
   ws.on('ping', () => {
     // Respond to ping to keep connection alive
     ws.pong();
   });
   
   ws.on('close', () => {
     console.log('Connection closed—reconnecting...');
     // Add reconnection logic here if needed
   });
   
   ws.on('error', (err) => {
     console.error('WebSocket connection error:', err);
   });
   

Step 3: Integrate with Angular #

For Angular, you can use a library like ngx-socket-io to simplify WebSocket integration:

1. Install the library:

   npm install ngx-socket-io
   

2. Configure the connection in your AppModule:

   import { NgModule } from '@angular/core';
   import { SocketIoModule, SocketIoConfig } from 'ngx-socket-io';
   
   // Replace with your DevTools values
   const socketConfig: SocketIoConfig = {
     url: 'wss://target-site.com/internal-realtime-endpoint',
     options: {
       extraHeaders: {
         'Origin': 'https://target-site.com',
         'Cookie': 'your-auth-cookie-from-devtools'
       }
     }
   };
   
   @NgModule({
     imports: [SocketIoModule.forRoot(socketConfig)],
     // ... other imports/declarations
   })
   export class AppModule {}
   

3. Use the socket in a component:

   import { Component, OnInit } from '@angular/core';
   import { Socket } from 'ngx-socket-io';
   
   @Component({
     selector: 'app-live-data',
     templateUrl: './live-data.component.html'
   })
   export class LiveDataComponent implements OnInit {
     liveData: any;
   
     constructor(private socket: Socket) {}
   
     ngOnInit(): void {
       // Listen for incoming messages
       this.socket.fromEvent('message').subscribe((data) => {
         this.liveData = data;
         console.log('Live data update:', this.liveData);
       });
   
       // Send the subscribe message to start receiving data
       this.socket.emit('subscribe', { channel: 'live-data-feed' });
     }
   }
   

Critical Things to Keep in Mind #

• Legality: Always check the site’s Terms of Service before scraping internal APIs. Many sites prohibit this, especially for commercial use, and you could face legal consequences or get your IP blocked.
• Stability: Internal WebSocket endpoints and data formats can change without warning (since they’re not public APIs). Your code might break unexpectedly, requiring you to re-reverse-engineer the connection.
• Authentication: If the site requires login, you’ll need to maintain a valid session. This might involve simulating a login flow (using tools like Puppeteer or Axios) to get fresh cookies/tokens periodically.
• Rate Limits: Sites may restrict the number of concurrent connections or data request rates. Too many failed attempts could result in your IP being blocked.

内容的提问来源于stack exchange，提问作者Adam Adamski