如何在Go语言中使用json.NewDecoder解析并控制大体积JSON流的控制流？

Hey there! Since you're dealing with a JSON file too big to load into memory entirely, using json.NewDecoder is exactly the right call—it streams content instead of loading everything at once. Let's break down how to tackle your specific scenarios, plus some best practices to make this smooth, especially as a new developer.

First, let's restate your JSON structure for clarity:

{
"key1" : [ "hundreds_of_nested_objects" ],
"key2" : [ "hundreds_of_nested_objects" ],
"unknown_unexpected_key" : [ "many_nested_objects" ],
........
"keyN" : [ n_objects ]
}

Scenario 1: Parse Only Specific Keys (e.g., key1 or keyN) #

Instead of processing every key in the root object, we can skip over ones we don't care about. Here's a focused example for targeting key1:

package main

import (
	"encoding/json"
	"io"
	"log"
	"os"
)

// Define your nested object structure (match fields to your actual JSON)
type NestedObject struct {
	ItemID   string `json:"item_id"`
	Value    int    `json:"value"`
	// Add other required fields here
}

func main() {
	file, err := os.Open("myfile.json")
	if err != nil {
		log.Fatalf("Failed to open file: %v", err)
	}
	defer file.Close()

	dec := json.NewDecoder(file)

	// Skip the opening '{' of the root object
	if _, err := dec.Token(); err != nil {
		log.Fatalf("Failed to read root token: %v", err)
	}

	// Iterate through each key-value pair in the root
	for dec.More() {
		key, err := dec.Token()
		if err != nil {
			log.Fatalf("Failed to read key: %v", err)
		}
		keyStr, ok := key.(string)
		if !ok {
			log.Fatal("Expected key to be a string")
		}

		if keyStr == "key1" {
			// Skip the ':' separator
			dec.Token()
			// Skip the opening '[' of the array
			dec.Token()

			// Decode each object in the array
			for dec.More() {
				var obj NestedObject
				if err := dec.Decode(&obj); err != nil {
					log.Fatalf("Failed to decode object: %v", err)
				}
				// Process the object (e.g., print, store)
				log.Printf("Processed key1 object: %+v", obj)
			}

			// Skip the closing ']' of the array
			dec.Token()
		} else {
			// Skip entire value for unwanted keys (saves memory/time)
			if err := dec.Skip(); err != nil {
				log.Fatalf("Failed to skip value: %v", err)
			}
		}
	}

	// Skip the closing '}' of the root object
	dec.Token()
}

This code ignores all keys except key1, so you only process the data you need.

Scenario 2: Find a Specific Nested Object Under a Target Key #

Let's say you want to find an object in key1 where ItemID equals "target_123". Modify the array decoding loop to add a check:

// Inside the key1 array processing block
for dec.More() {
	var obj NestedObject
	if err := dec.Decode(&obj); err != nil {
		log.Fatalf("Failed to decode object: %v", err)
	}

	// Check if this is the object we're looking for
	if obj.ItemID == "target_123" {
		log.Printf("Found target object: %+v", obj)
		// Break here if you only need the first match
		// break
	}
}

For multiple target keys, just add additional if keyStr == "keyN" blocks with similar logic.

Scenario 3: Export Specific Key Content to Another File #

To export the entire key1 array or filtered objects to a new JSON file, pair json.NewDecoder with json.NewEncoder:

Export the full key1 array #

func main() {
	inputFile, err := os.Open("myfile.json")
	if err != nil {
		log.Fatalf("Failed to open input: %v", err)
	}
	defer inputFile.Close()

	outputFile, err := os.Create("key1_export.json")
	if err != nil {
		log.Fatalf("Failed to create output: %v", err)
	}
	defer outputFile.Close()

	enc := json.NewEncoder(outputFile)
	enc.SetIndent("", "  ") // Optional: for readable, indented JSON

	dec := json.NewDecoder(inputFile)

	// Skip root '{'
	dec.Token()

	for dec.More() {
		key, _ := dec.Token()
		keyStr := key.(string)

		if keyStr == "key1" {
			// Write opening '[' to output
			outputFile.WriteString("[\n")

			// Skip ':' and '[' in input
			dec.Token()
			dec.Token()

			firstObj := true
			for dec.More() {
				var rawObj json.RawMessage // Preserve original JSON structure
				dec.Decode(&rawObj)

				if !firstObj {
					outputFile.WriteString(",\n")
				} else {
					firstObj = false
				}

				enc.Encode(rawObj)
			}

			// Write closing ']' to output
			outputFile.WriteString("\n]")
			break // Stop processing once we're done with key1
		} else {
			dec.Skip()
		}
	}
}

Export only filtered objects from key1 #

Adjust the array loop to add your filter before encoding:

firstObj := true
for dec.More() {
	var obj NestedObject
	dec.Decode(&obj)

	// Only export objects where Value is greater than 500
	if obj.Value > 500 {
		if !firstObj {
			outputFile.WriteString(",\n")
		} else {
			firstObj = false
		}
		enc.Encode(obj)
	}
}

Best Practices for Large JSON Parsing #

• Use dec.Skip() aggressively: For keys/values you don't need, Skip() tells the decoder to jump over the entire value without parsing it—this saves huge amounts of memory and CPU.
• Keep structs minimal: Only define fields you need in your NestedObject struct. Go ignores extra JSON fields automatically, and smaller structs decode faster.
• Buffer I/O operations: Wrap files with bufio.Reader/bufio.Writer to reduce system calls (faster for large files):

  import "bufio"
  
  // Input
  bufInput := bufio.NewReader(inputFile)
  dec := json.NewDecoder(bufInput)
  
  // Output
  bufOutput := bufio.NewWriter(outputFile)
  enc := json.NewEncoder(bufOutput)
  defer bufOutput.Flush() // Don't forget to flush when done!
  

• Test with small samples: Before running on the full 10GiB file, test your code with a trimmed-down version of the JSON to catch bugs early.
• Handle errors gently: In production, avoid log.Fatal() unless it's a critical failure—consider returning errors and letting the caller decide how to handle partial data.

内容的提问来源于stack exchange，提问作者arif