基于Express/Node.js的储罐液位变化检测技术问询

Hey there! Let's break down each of your questions based on building a tank level dashboard with Express/Node.js—this is a common industrial IoT use case, so I’ve got practical, actionable insights for you:

You’ve got a few solid options depending on whether you need real-time detection or post-hoc analysis:

• Sliding Window Slope Detection: Calculate the linear regression slope over a small window of recent samples (e.g., 3-5 data points, 15-25 minutes). A consistent positive slope signals filling, negative signals emptying. This is lightweight for real-time use.
• Piecewise Linear Regression (PLR): Split your time-series data into segments with constant slopes. Tools like the regression npm package can help identify breakpoints where the slope changes—perfect for post-hoc daily report generation to cleanly map full fill/empty events.
• Threshold-Based Detection with Hysteresis: Set upper/lower thresholds for rate-of-change (e.g., "if level increases by >X cm in 5 minutes, trigger fill event"). Add hysteresis (wait for 2 consecutive samples to confirm the change) to avoid noise triggers.
• Kalman Filter + Change Point Detection: First smooth noisy data with a Kalman filter, then use methods like the Cumulative Sum (CUSUM) test to spot abrupt shifts in the smoothed level signal. Great for noisy sensor environments.

There are two main integration paths, depending on whether you’re handling real-time or batch processing:

• Real-Time Processing (During Data Ingestion):
  • Add a middleware function after parsing incoming sensor data but before storing it in the database. Use this middleware to run your slope/threshold checks.
  • To avoid blocking your Express server’s event loop, offload heavy calculations (like PLR or Kalman filtering) to a worker thread or a job queue (e.g., BullMQ). Store temporary event state (e.g., "fill in progress") in Redis or your database.
• Batch Processing (For Daily Reports):
  • Write a standalone Node.js script that queries your database for all samples from the previous day. Use this script to run PLR or full dataset slope analysis to finalize events.
  • Schedule the script to run automatically at the end of each day using node-cron or node-schedule.

I’d recommend a hybrid approach for accuracy and efficiency:

• Real-Time Processing at Ingestion: Do lightweight checks (sliding window slope, basic thresholding) to mark potential event start/end points in your database (e.g., add event_status and event_id fields). This lets you trigger real-time alerts if needed.
• Daily Batch Processing: At the end of the day, run a full analysis on all daily data to clean up false positives from noise, merge overlapping events, and calculate precise event durations/volume changes. This is where PLR or CUSUM will shine—you get a holistic view of the day’s activity to generate your final report.

Noise spikes will wreck your event detection, so you need to clean data before processing or storing it (or store both raw and cleaned data for auditing). Here’s how to handle those random spikes:

• Sliding Window Mean/Median Filter: Replace each sample with the mean or median of the current sample plus the previous 2-3 samples. Median filtering is especially good at eliminating isolated outliers.
• Rate-of-Change Thresholding: If a sample’s level change from the previous one is way outside your expected normal range (e.g., >10cm in 5 minutes when typical fill rate is 2cm/5min), flag it as noise. You can either discard it or replace it with an interpolated value between the previous and next valid sample.
• Kalman Filter: For more persistent sensor noise, implement a simple Kalman filter (use the kalman-filter npm package or roll your own basic version) to smooth the signal before further processing.

The key here is using a state machine paired with slope validation to detect clean transitions:

1. Maintain an Event State: Track whether you’re in "idle", "filling", or "emptying" state (store this in Redis or your database for persistence).
2. Slope Transition Check: When a sliding window slope switches from positive (fill) to negative (empty), don’t immediately mark the fill event as done—wait for 1-2 consecutive samples (5-10 minutes) to confirm the negative slope is consistent (this filters out noise-induced false transitions).
3. Update Event Metadata: Once the transition is confirmed, close the current fill event (record end time and total volume increase) and start a new empty event. Use your cleaned, smoothed data to calculate precise duration and volume changes.

Based on your note about stable slopes for ~15 minutes and reduced total fill volume, here’s a practical approach:

1. Establish Baseline Metrics: First, analyze historical normal fill events to define:
   • Typical fill slope range (e.g., +1.8 to +2.2 cm/5min)
   • Expected total volume increase for a full fill (e.g., 500L ± 50L)
2. Flag Suspect Events: When you detect a fill slope that stays within the baseline range for 15 minutes, but the total volume increase is consistently below the expected baseline (e.g., only 300L after 30 minutes), mark this as a "mixed fill/empty" event.
3. Validate with Persistence: Confirm that the level stays stable (slope near 0) after the initial 15-minute period—this indicates inflow and outflow are balanced.
4. Report Differently: In your daily report, separate these mixed events from normal fill/empty operations, noting the reduced volume and balanced phase.

内容的提问来源于stack exchange，提问作者Mejezza