Apache NiFi技术问题：校验evaluateJsonPath列表元素与筛选含tech标签的Twitter文件

Hey there! Let's break down your two Apache NiFi questions with practical, step-by-step solutions that you can implement right away.

EvaluateJsonPath excels at extracting JSON data, but it doesn’t handle validation on its own. To check if elements in a list meet your criteria, pair it with other NiFi processors like this:

• Step 1: Extract the target list with EvaluateJsonPath
  Configure the EvaluateJsonPath processor:

  • Set Destination to flowfile-attribute (to store the list as an attribute for easy access)
  • Add a new property (e.g., target-list) with a JSONPath expression to pull your list. For example, to get the hashtags list from your Twitter JSON:

    $.entities.hashtags
    

• Step 2: Validate elements using RouteOnAttribute
  Add a RouteOnAttribute processor and create a routing rule (e.g., has-valid-element). Use NiFi's Expression Language to check if the list contains your desired element:

  • For a simple check (looking for the exact string "text":"tech" in the list attribute):

    ${target-list:contains('"text":"tech"')}
    

  • For a more precise check (using JSONPath directly in the expression):

    ${jsonPath($, '$.entities.hashtags[?(@.text == "tech")]'):isEmpty():not()}
    

  Route FlowFiles that pass the check to your desired relationship (e.g., valid), and handle others as needed.

• Alternative: Use FilterRecord for structured data
  If you’re working with consistent JSON structures, FilterRecord is more efficient. Configure it with a JsonTreeReader (enable Infer Schema to auto-detect your JSON structure) and use this RecordPath expression to filter:

  /entities/hashtags[text == 'tech']
  

  Only FlowFiles with matching list elements will be retained.

For your specific Twitter JSON example, here are two reliable ways to filter FlowFiles that include the "tech" hashtag:

Method 1: EvaluateJsonPath + RouteOnAttribute (Lightweight) #

This is the go-to for straightforward filtering:

1. Set up EvaluateJsonPath
   • Set Destination to flowfile-attribute
   • Add a property like has-tech-tag with this JSONPath expression (it returns matching hashtags or an empty value if none exist):

     $.entities.hashtags[?(@.text == "tech")]
     

2. Configure RouteOnAttribute
   • Create a rule named include with this expression to check for non-empty results:

     ${has-tech-tag:notEmpty()}
     

   • Route matching FlowFiles to include and handle non-matching ones via unmatched or discard them.

Method 2: FilterRecord (Batch-Friendly) #

Ideal for large volumes of structured JSON:

1. Configure JsonTreeReader
   • Enable Infer Schema so NiFi automatically detects your Twitter JSON structure (no manual schema needed).
2. Set up FilterRecord
   • Select your JsonTreeReader as the Record Reader
   • Choose JsonRecordSetWriter as the Record Writer to keep the original JSON format
   • Enter this RecordPath in Filter Record Path to check for the "tech" hashtag:

     exists(/entities/hashtags[text == 'tech'])
     

   • Only FlowFiles with the "tech" hashtag will be output.

内容的提问来源于stack exchange，提问作者Anil