LUIS训练集与查询中句号处理的最佳实践及分词机制咨询

Great question—this is a common point of confusion when working with LUIS, especially since punctuation can subtly shift intent recognition results like you’ve noticed with "Hi. I'd like to order." vs "Hi I'd like to order".

Let’s break down how LUIS processes periods specifically:

• Tokenization Behavior: By default, LUIS treats periods as a separate token during the initial processing stage, but context matters a lot:
  • If the period is part of a common abbreviation (like "Mr." or "U.S.A."), LUIS will usually keep it attached to the preceding word as a single token, since it recognizes these as semantic units.
  • For standalone periods (like the one after "Hi" in your example), it gets split into its own distinct token. But here’s the key: during the feature extraction phase, LUIS might downplay or ignore standalone punctuation tokens if they don’t add meaningful semantic value.
• Cleanup Scenarios: LUIS does perform selective punctuation cleanup, but it’s not a one-size-fits-all rule. Standalone periods (and other non-meaningful punctuation) are often removed before intent classification, but this isn’t always consistent. Sometimes the mere presence of that period can alter how the model interprets phrase boundaries, which explains why you see differing intent results.
• Why Intent Recognition Differs: The split between "Hi." and "Hi" changes the token sequence the model analyzes. Even if the period is eventually ignored, the initial tokenization can shift how the model weights surrounding words. For example, "Hi." as a separate token pair might signal a clear greeting followed by a distinct request, whereas "Hi" merged with the next phrase could be interpreted as a more continuous utterance.

If you want to standardize this behavior, you can adjust LUIS’s tokenizer settings (in the Language Understanding portal under "Manage > Settings") or preprocess your utterances to normalize punctuation before sending them to LUIS—either removing standalone periods consistently or keeping them intact across all inputs.

内容的提问来源于stack exchange，提问作者Amotz Hoshen