如何使用NiFi的PutHbaseJson/PutHbaseCell插入多条JSON数据到HBase？

Hey there! Let's walk through exactly how to get those multiple JSON records into HBase using NiFi's two HBase processors. I'll start with the one you've already experimented with—PutHbaseCell—since you have sample data ready, then cover the more JSON-friendly PutHbaseJson for comparison.

Using PutHbaseCell (Your Current Processor) #

PutHbaseCell works best when you need explicit control over HBase columns, but it requires splitting your multi-JSON input into individual records first. Here's the step-by-step workflow:

1. Split Your Multi-JSON FlowFile into Single Records

   • If your input is a comma-separated list of JSON objects (like your sample):
     • First, use a ReplaceText processor to wrap the content in an array:
       • Set Search Value to ^ (start of content) and Replacement Value to [
       • Add a second ReplaceText processor: set Search Value to $ (end of content) and Replacement Value to ]
     • Then add a SplitJson processor with JsonPath Expression set to $.*—this will split the array into individual FlowFiles, each containing one JSON object.
   • If your input is already a valid JSON array ([{"id":"1"},{"id":"2"}]), skip the ReplaceText steps and go straight to SplitJson.

2. Extract JSON Fields as NiFi Attributes

   • Add an EvaluateJsonPath processor to pull out the fields you need for HBase:
     • Create these user-defined properties:
       • row_id → $.id (this will be your HBase row key)
       • name_val → $.name
       • path_val → $.path
   • Set Destination to flowfile-attribute so these values are stored as FlowFile attributes.

3. Configure PutHbaseCell

   • Fill in the core settings:
     • HBase Configuration Resources: Path to your hbase-site.xml file (so NiFi can connect to your HBase cluster)
     • Table Name: The name of your target HBase table (e.g., product_categories)
     • Row ID: Use the attribute we extracted: ${row_id}
   • Under the Columns section, add entries for each field you want to insert:
     • For the name field:
       • Column Family: info (replace with your actual column family)
       • Column Qualifier: name
       • Value: ${name_val}
     • For the path field:
       • Column Family: info
       • Column Qualifier: path
       • Value: ${path_val}
   • Adjust Batch Size (default 1000) if you want to tune how many records are inserted in one batch.

Using PutHbaseJson (Simpler for JSON Data) #

If your JSON structure is consistent, PutHbaseJson is more streamlined—it maps JSON fields directly to HBase columns without needing to extract attributes first. Here's how to set it up:

1. Split Multi-JSON into Single Records

   • Follow the same SplitJson (and ReplaceText if needed) steps as above to get one JSON object per FlowFile.

2. Configure PutHbaseJson

   • Core settings:
     • HBase Configuration Resources: Same as above, path to hbase-site.xml
     • Table Name: Your target HBase table
     • Row ID JsonPath: $.id (tells the processor which JSON field to use as the HBase row key)
   • Map JSON fields to HBase columns:
     • Option 1 (Single Column Family): Set Column Family to your desired family (e.g., info), and the processor will automatically map every JSON field to a column qualifier with the same name (so name → info:name, path → info:path).
     • Option 2 (Custom Mappings): Use the Column Mappings property to specify exact mappings, like:

       name:info:product_name, path:info:category_path
       

       Format: json-field-name:hbase-column-family:hbase-column-qualifier

Key Notes to Avoid Issues #

• Permissions: Make sure the NiFi service account has write access to your HBase table.
• Debugging: Use a LogAttribute processor after EvaluateJsonPath/SplitJson to verify that attributes or split records are correct before sending to HBase.
• Batch Performance: Adjust the Batch Size in both processors based on your cluster's capacity—larger batches can improve throughput but may increase latency.

内容的提问来源于stack exchange，提问作者Umang