使用Data Factory管道填充Azure Search索引技术咨询

Hey there! Since you're new to Azure Search and Data Factory, let's break down exactly how to set up a pipeline that populates your Azure Search index from your Azure SQL database. I'll walk you through each step clearly:

Before jumping into Data Factory, make sure you have:

• An Azure Search service already created in your Azure subscription
• A matching Search index where you want to store the drug data. The index's fields should align with your [transformed].[drug] table structure (for example, your table has a DrugSearchId uniqueidentifier column—your index needs a corresponding field, like Edm.Guid or Edm.String).

You already have your Azure SQL linked service set up, so next we need to connect Data Factory to your Azure Search service:

• Go to your Data Factory resource, then navigate to Manage > Linked Services > New
• Search for and select Azure Cognitive Search (note: older versions might label this "Azure Search")
• Fill in the config:
  • Give it a name like AzureSearchLinkedService
  • Pick your Azure Search service instance from the dropdown
  • Paste in the admin key (you can grab this from your Azure Search service's Keys page in the Azure portal)
  • Test the connection to confirm it works, then save

Now create a dataset that points directly to your target Search index:

• Head to Author > Datasets > New
• Choose the Azure Cognitive Search type and click Continue
• Configure the dataset:
  • Name it something like DrugSearchIndexDataset
  • Select the AzureSearchLinkedService you just created
  • Pick your target Search index from the list
  • Save the dataset

This is the core part—we'll create a pipeline that copies data from your SQL table to the Search index:

1. Go to Author > Pipelines > New Pipeline
2. Drag a Copy Data activity from the activity panel onto the canvas
3. Configure the Source tab:
   • Select your existing Drug SQL dataset (here's the config you shared, for reference):

     { 
       "name": "Drug", 
       "properties": { 
         "published": false, 
         "type": "AzureSqlTable", 
         "linkedServiceName": "AzureSqlLinkedService", 
         "typeProperties": { "tableName": "[transformed].[drug]" }, 
         "availability": { "frequency": "Minute", "interval": 15 }, 
         "external": true, 
         "policy": {} 
       } 
     }
     

   • Since your source dataset is set to refresh every 15 minutes, you can stick with full copy or set up incremental copy later if you need to sync only new/updated data.
4. Configure the Sink tab:
   • Select your DrugSearchIndexDataset
   • Under Settings, choose your write behavior:
     • Upsert: Recommended—updates existing documents if DrugSearchId matches, inserts new ones if not
     • Insert: Only adds new documents (won't update existing ones)
     • Delete: Removes documents matching the key
   • Double-check the field mappings: Data Factory usually auto-matches fields, but confirm that DrugSearchId and other columns line up with your Search index fields correctly.
5. Save the pipeline when you're done.

• Click the Debug button at the top of the canvas to run a test run. Check the output to make sure data is flowing correctly to your Search index.
• If the test works, set up a trigger (matching your source's 15-minute frequency) to automate the sync. Go to Add trigger > New/Edit and configure a schedule trigger with the same 15-minute interval.

• For your DrugSearchId (uniqueidentifier), Azure Search supports both Edm.Guid and Edm.String field types—use whichever fits your use case better.
• If you have a large dataset, consider adding a timestamp column to your SQL table (like LastUpdated) and setting up incremental copy in the source to only pull data that's changed since the last run. This will make your pipeline more efficient.
• Make sure your Search index fields allow null values if your SQL table has any nullable columns—otherwise the copy activity might fail.

内容的提问来源于stack exchange，提问作者user3603308