针对Chat Completion API重复发送含微小调整的大Prompt的低成本替代方案咨询

Hey there! Great question—this is a super common pain point when working with large context prompts and iterative tweaks. Let’s break down the practical, actionable solutions you can use to cut down on token costs:

• Lean on incremental conversation context management
  If your large prompt has a fixed core context plus small tweaks, split it up: send the full core context as a system message in your first API call. For subsequent calls, only send the minor adjustments and your new user input, not the entire core context again. Most modern chat models will retain the system message context across follow-up calls (as long as you keep the conversation thread consistent with message roles). Just note that some models have limits on system message length, so this works best if your core context fits within those bounds.

• Use a vector database for contextual retrieval
  Split your big context into smaller, semantically meaningful chunks, then convert those chunks into vector embeddings and store them in a vector database. When you need to make a tweak, first run a semantic search to pull only the chunks relevant to your adjustment, then combine those with your tweak instruction into a concise prompt. This way, you never send the full context—only the parts the model actually needs for that specific call, drastically reducing token usage.

• Leverage tool/function calling capabilities
  If your model supports tool calling, wrap your fixed large context into a custom "information lookup" tool. Instead of feeding all the context upfront, define the tool in your API call, and let the model request specific parts of the context when it needs them. For follow-up tweaks, you only need to send the tweak instruction; the model will pull the necessary context via the tool instead of you repeating it every time.

• Compress your core context upfront
  If your large context has redundant information, use a lightweight model to generate a condensed summary of the core points first. Then, for every subsequent tweak, you only need to send this condensed summary plus your adjustment. The initial compression uses some tokens, but every follow-up call will be way cheaper—especially if your tweaks are only targeting small sections of the original context.

• Build your own session storage
  If the API you’re using doesn’t offer built-in session retention, set up your own backend to store the full conversation history (including the large core context). For each follow-up call, only send the new tweak instruction and any incremental changes, then have your backend append that to the stored history before sending the full thread to the API. This way, you don’t have to manually repeat the big context in every request—your backend handles the heavy lifting.

备注：内容来源于stack exchange，提问作者Ishaan Sejwal