能否将C#开发的Storm拓扑部署至非Azure的本地/自建Storm集群？

Absolutely! You absolutely can deploy your C#-based Storm topologies to a non-Azure Storm cluster (including a Dockerized one) — the Azure Data Lake Tools/Visual Studio setup doesn’t lock you into HDInsights. Here’s a step-by-step breakdown to make it work:

1. Understand the underlying SDK #

The C# Storm components you built using Azure Data Lake Tools rely on the Microsoft.SCP.Net.SDK (a NuGet package), which is fully compatible with standard Apache Storm clusters, not just Azure HDInsights. This SDK bridges C# code to Storm’s Java-based runtime via Thrift, so your topology can run anywhere Storm does.

2. Prepare your Dockerized Storm cluster #

First, get a working standard Storm cluster up in Docker:

• Use an official or trusted Storm Docker image (e.g., apache/storm or Bitnami’s Storm image).
• Ensure the cluster’s Nimbus node is reachable from your development machine (map port 6627, Storm’s Nimbus port, if running locally).
• Critical: Install the appropriate .NET Runtime on all Storm worker nodes. If your topology uses .NET Core/.NET 6+, add the runtime to your Docker image (e.g., for a Linux-based Storm image, install the .NET 6 Runtime via apt-get or yum). For .NET Framework, you’d need a Windows-based Storm container (less common, but possible).

3. Package your C# topology #

From your Visual Studio project:

• Build your topology in Release mode.
• Collect all output files: your topology’s DLL/exe, Microsoft.SCP.Net.SDK.dll, and any other third-party DLLs your code depends on (e.g., Newtonsoft.Json).
• Package these files into a ZIP archive (keep the directory structure intact if your code uses relative paths).

4. Submit the topology to your cluster #

You have two main options to submit:

Option A: Use the SCP SDK submission tool #

The Microsoft.SCP.Net.SDK includes a command-line tool (scp.exe) for submitting topologies. It’s usually located in your project’s output directory or the NuGet package’s tools folder. Run this command:

scp.exe submit --nimbus <docker-nimbus-ip>:6627 --topology-name MyIoTSensorTopology --assembly ./MyTopology.dll --class MyNamespace.MyTopologyClass

Replace placeholders with your cluster’s Nimbus IP, topology name, DLL path, and fully qualified topology class name.

Option B: Use Storm’s native storm submit command #

If you prefer using Storm’s built-in tooling, you can submit your ZIP package directly:

storm submit --nimbus <docker-nimbus-ip>:6627 --jar ./MyTopologyPackage.zip --class MyNamespace.MyTopologyClass

Storm will handle transferring the package to worker nodes and launching the .NET-based components.

5. Troubleshooting tips #

• Network issues: Verify your development machine can ping the Docker Nimbus node and reach port 6627 (use telnet or nc to test).
• Missing dependencies: Double-check that all required DLLs are included in your ZIP package — missing files will cause worker nodes to fail loading your topology.
• Runtime mismatches: Ensure the .NET Runtime version on worker nodes matches the version you used to compile your topology (e.g., .NET 6 compiled code won’t run on .NET 5 Runtime).

内容的提问来源于stack exchange，提问作者Dovic