Great question—this is a super common (but tricky!) scenario for IoT, mobile, or edge devices. I’ve implemented this exact flow across several production projects, so here’s a battle-tested, optimal approach:

The goal is to guarantee data isn’t lost offline, and ensure it gets uploaded reliably once connectivity returns. Let’s break it down step by step:

1. Choose the Right Local Storage #

Pick a storage option that fits your device’s constraints, with a focus on data durability:

• Embedded/IoT devices: Use lightweight databases like SQLite (supports transactions, easy querying) or structured log files (simpler, but harder to manage state). Avoid volatile RAM—always write to persistent storage like Flash or SD cards.
• Mobile apps: Use platform-native solutions like Room (Android) or Core Data (iOS), or cross-platform options like SQLite or Realm.
• Desktop/Server edge: SQLite, LevelDB, or even a lightweight KV store work well.

Critical note: Always use atomic writes (e.g., database transactions) when saving data locally. This prevents partial writes if the device suddenly powers off.

2. Build an Offline Queue with Metadata #

Don’t just dump raw data locally—add metadata to track each entry’s lifecycle. Your queue entries should include:

• A unique ID (UUID works great) for idempotency (more on this later)
• The raw data payload (JSON, binary, etc.)
• Upload status (pending, uploading, failed, success)
• Retry count
• Timestamp of creation

This metadata lets you avoid duplicate uploads, retry failed attempts, and clean up successful entries later.

3. Detect Connectivity & Trigger Uploads #

Implement a way to check when the device is back online:

• Mobile: Listen for system network change events (e.g., Android’s CONNECTIVITY_ACTION broadcast, iOS’s NWPathMonitor).
• Embedded/IoT: Periodically ping a reliable endpoint (like your server or a public DNS) to check connectivity.
• All platforms: Run the upload task in a background thread/worker to avoid blocking your main application flow.

Once online, fetch all pending or failed entries from your local queue and start uploading them one by one (or in batches, if your server supports it).

4. Smart Retry & Error Handling #

Blindly retrying failed uploads will waste resources and hammer your server. Instead:

• Use exponential backoff: Start with short delays (10s), then double the wait time each retry (20s, 40s, etc.) until you hit a max retry limit (e.g., 3-5 attempts).
• Differentiate error types:
  • Skip retries for client-side errors (4xx status codes like 400 Bad Request)—these mean your data is invalid, and retrying won’t help. Mark these as failed for manual review.
  • Retry for server-side errors (5xx) or network timeouts—these are transient issues that may resolve when connectivity improves.
• Mark entries as uploading before sending them to prevent concurrent upload attempts of the same data.

5. Enforce Idempotency on the Server #

Even with perfect client-side logic, network flakiness can cause scenarios where your server receives data but the client doesn’t get the success response. To avoid duplicate processing:

• Include the unique ID from your local queue in every upload request.
• On the server, first check if that ID has already been processed. If yes, return a success response immediately without reprocessing the data.
• If not, process the data and store the ID to track it.

Quick Example (Python + SQLite) #

Here’s a simplified snippet to illustrate the flow:

import sqlite3
import json
import uuid
import requests
import time

# Initialize local queue table (run once)
def init_db():
    with sqlite3.connect('offline_queue.db') as conn:
        cursor = conn.cursor()
        cursor.execute('''
            CREATE TABLE IF NOT EXISTS data_queue (
                id TEXT PRIMARY KEY,
                payload TEXT NOT NULL,
                status TEXT DEFAULT 'pending',
                retry_count INTEGER DEFAULT 0,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        ''')
        conn.commit()

# Save data to local queue when offline
def save_offline_data(payload):
    unique_id = str(uuid.uuid4())
    with sqlite3.connect('offline_queue.db') as conn:
        cursor = conn.cursor()
        cursor.execute(
            "INSERT INTO data_queue (id, payload) VALUES (?, ?)",
            (unique_id, json.dumps(payload))
        )
        conn.commit()

# Upload pending data when online
def upload_pending_data():
    with sqlite3.connect('offline_queue.db') as conn:
        cursor = conn.cursor()
        # Fetch eligible entries to upload
        cursor.execute('''
            SELECT id, payload FROM data_queue 
            WHERE status IN ('pending', 'failed') AND retry_count < 3
        ''')
        rows = cursor.fetchall()

        for data_id, payload in rows:
            try:
                # Mark as uploading to prevent duplicates
                cursor.execute(
                    "UPDATE data_queue SET status = 'uploading' WHERE id = ?",
                    (data_id,)
                )
                conn.commit()

                # Send to server with unique ID for idempotency
                response = requests.post(
                    "https://your-server-endpoint.com/upload",
                    json={"request_id": data_id, "data": json.loads(payload)}
                )
                response.raise_for_status()

                # Mark as successful
                cursor.execute(
                    "UPDATE data_queue SET status = 'success' WHERE id = ?",
                    (data_id,)
                )
                conn.commit()

            except requests.exceptions.RequestException as e:
                # Update retry count and mark as failed
                cursor.execute('''
                    UPDATE data_queue 
                    SET status = 'failed', retry_count = retry_count + 1 
                    WHERE id = ?
                ''', (data_id,))
                conn.commit()

                # Exponential backoff
                retry_num = cursor.execute(
                    "SELECT retry_count FROM data_queue WHERE id = ?",
                    (data_id,)
                ).fetchone()[0]
                time.sleep(10 * (2 ** retry_num))

# Clean up old successful entries to save space
def cleanup_old_data(days=7):
    with sqlite3.connect('offline_queue.db') as conn:
        cursor = conn.cursor()
        cursor.execute(f'''
            DELETE FROM data_queue 
            WHERE status = 'success' 
            AND created_at < DATETIME('now', '-{days} days')
        ''')
        conn.commit()

Bonus Critical Tips #

• Resource Management: For memory-constrained devices, regularly clean up success entries to avoid filling up storage.
• Encryption: If handling sensitive data, encrypt payloads before storing them locally, and always use HTTPS for uploads.
• Large Files: For big payloads (like videos), use chunked uploads and track which chunks have been uploaded locally to resume progress after interruptions.
• Concurrency Safety: If multiple threads/processes access the queue, use database locks or transactions to prevent race conditions.

内容的提问来源于stack exchange，提问作者jon