如何在Heron中结合Local组件与默认算法使用自定义调度器？

Great to hear you've built your CustomScheduler already! Let's break down the remaining steps to get it working with LocalLauncher, LocalUploader, and the default packing algorithm, plus how to verify everything is running correctly.

1. Complete the Configuration Files #

You mentioned modifying conf/local/scheduler.yaml, but let's make sure all related configs are set properly:

• Launcher Configuration (conf/local/launcher.yaml)
  Ensure this file specifies the LocalLauncher as the launcher class. Add or update the line:

  launcher_class: org.apache.heron.launcher.local.LocalLauncher
  

• Uploader Configuration (conf/local/uploader.yaml)
  Similarly, set the LocalUploader here with its full class path:

  uploader_class: org.apache.heron.uploader.local.LocalUploader
  

• Packing Configuration (conf/local/packing.yaml)
  Since you want to use the default packing algorithm, either leave this file as-is (it uses the default implementation by default) or explicitly specify it to avoid ambiguity:

  packing_class: org.apache.heron.packing.roundrobin.RoundRobinPacking
  

• Double-check Your scheduler.yaml
  Confirm your CustomScheduler's full class name is correctly set here (replace com.your.package.CustomScheduler with your actual class path):

  scheduler_class: com.your.package.CustomScheduler
  

2. Deploy Your CustomScheduler Jar #

Heron needs to load your CustomScheduler class, so pick one of these deployment options:

• Global Deployment: Copy your compiled CustomScheduler jar into Heron's installation directory under lib/—this makes it available for all topologies submitted to the local cluster.
• Per-Topology Deployment: When submitting a topology, use the --libjars flag to specify the path to your jar, like:

  heron submit local --libjars /path/to/your/custom-scheduler.jar ...
  

3. Submit a Test Topology for Verification #

Use Heron's built-in example topology to test your setup—it's simple and perfect for validation:

1. Submit the topology:

   heron submit local --config-path conf/local/ com.twitter.heron.examples.ExclamationTopology ExclamationTest
   

2. Check if the topology is running:
   Run this command to list all topologies on the local cluster:

   heron list local
   

   You should see ExclamationTest listed with a RUNNING state if everything works.

3. Inspect Logs to Confirm CustomScheduler is Active:
   Heron stores logs locally under ~/.herondata/topologies/local/<your-username>/ExclamationTest/logs/.

   • Open the scheduler logs at scheduler/scheduler.log—look for lines indicating your CustomScheduler is being used (e.g., Using scheduler class: com.your.package.CustomScheduler or any custom log statements you added to your scheduler code).
   • Check launcher and uploader logs in their respective directories to ensure LocalLauncher and LocalUploader are functioning as expected.

4. Troubleshooting Common Issues #

• ClassNotFoundException: If you see this error, double-check that your CustomScheduler jar is in the correct location or that the --libjars path is accurate. Also verify the full class name in scheduler.yaml has no typos.
• Topology Fails to Launch: Check the launcher logs first—they'll show if there's an issue with starting the topology. Also ensure your CustomScheduler implements all required methods from the IScheduler interface correctly.
• Unexpected Packing Behavior: Confirm the packing_class in packing.yaml is set to the default algorithm you intend to use.

内容的提问来源于stack exchange，提问作者Yitian Zhang