curl与Apache Camel CouchDB Producer上传JSON行为差异及批量存储方案

I get exactly what you're facing—by default, the Apache Camel CouchDB Producer treats the entire JSON file as a single document, which isn't what you want when you have a docs array with pre-defined _id values. Let's fix this with two practical approaches, depending on whether you want efficient batch processing or need to handle each document individually.

Approach 1: Use CouchDB's _bulk_docs Endpoint (Most Efficient) #

This mirrors the curl command you're already using, sending the entire {"docs": [...]} payload directly to CouchDB's bulk API. It’s the fastest option because it minimizes HTTP requests.

Step-by-Step Setup: #

1. Adjust your Camel route to target the _bulk_docs path and set the required HTTP headers:

   import org.apache.camel.Exchange;
   import org.apache.camel.component.couchdb.CouchDbConstants;
   import org.apache.camel.builder.RouteBuilder;
   
   public class CouchDbBulkRoute extends RouteBuilder {
       @Override
       public void configure() throws Exception {
           from("file:/home/r2/Desktop/couchInput?noop=true&fileName=file1.json")
               // Set Content-Type to match JSON payload requirement
               .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
               // Specify POST method since _bulk_docs requires it
               .setHeader(CouchDbConstants.METHOD, constant(CouchDbMethod.POST))
               // Route to the _bulk_docs endpoint of your "order" database
               .to("couchdb:http://localhost:5984/order?path=_bulk_docs");
       }
   }
   

2. How it works:

   • The route reads your JSON file as-is (with the docs array intact).
   • It sends the entire payload to http://localhost:5984/order/_bulk_docs via POST, just like your curl command.
   • CouchDB processes each document in the docs array automatically:
     • If a document has an existing _id and matching _rev, it updates that document.
     • If the _id doesn’t exist, it creates a new document with that ID.

Approach 2: Split the docs Array (For Per-Document Processing) #

If you need to modify, validate, or enrich each document individually before sending it to CouchDB, split the docs array and process each entry separately.

Step-by-Step Setup: #

1. Create a Processor Bean to parse the JSON and extract the docs array:

   import com.fasterxml.jackson.databind.JsonNode;
   import com.fasterxml.jackson.databind.ObjectMapper;
   import org.apache.camel.Exchange;
   import java.io.InputStream;
   import java.util.Iterator;
   
   public class ProcessBean {
       public void process(Exchange exchange) throws Exception {
           InputStream jsonInputStream = exchange.getIn().getBody(InputStream.class);
           ObjectMapper mapper = new ObjectMapper();
           
           // Parse the root JSON object and extract the docs array
           JsonNode rootNode = mapper.readTree(jsonInputStream);
           JsonNode docsArray = rootNode.get("docs");
           
           // Pass array elements to the next step for splitting
           if (docsArray.isArray()) {
               Iterator<JsonNode> docsIterator = docsArray.elements();
               exchange.getOut().setBody(docsIterator);
           }
       }
   }
   

2. Update your Camel route to split the array and send each document:

   import org.apache.camel.Exchange;
   import org.apache.camel.builder.RouteBuilder;
   
   public class CouchDbSplitRoute extends RouteBuilder {
       @Override
       public void configure() throws Exception {
           from("file:/home/r2/Desktop/couchInput?noop=true&fileName=file1.json")
               .bean(ProcessBean.class)
               // Split the docs array into individual documents
               .split(body())
                   .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
                   // Send each document to CouchDB (uses _id/_rev if present)
                   .to("couchdb:http://localhost:5984/order")
               .end(); // Close the split block
       }
   }
   

Key Notes: #

• Dependencies: Ensure you have the Jackson JSON library in your project (it’s often included with Camel, but double-check your build configuration).
• Error Handling: Both approaches let you capture CouchDB’s response (e.g., success/failure status for each document) by accessing the Exchange’s body after the to() step. Add onException() blocks to retry failed documents if needed.
• Component Version: Use a recent version of the Camel CouchDB component (2.20+) to support these features reliably.

内容的提问来源于stack exchange，提问作者Rajat.r2