Eclipse运行StormCrawler时Custom parsefilter.json文件缺失的技术咨询

Let’s work through this error step by step—this is a common issue with resource loading in StormCrawler/Maven projects, and there are several key areas to check:

1. Verify the file's location in your project #

StormCrawler expects parsefilter.json to be accessible via the classpath. For Maven projects, the standard place to put this file is src/main/resources. This ensures Maven automatically includes it in the final JAR's root classpath during packaging. If you’re using a custom subdirectory under resources, you’ll need to reference the full path (e.g., /filters/parsefilter.json) when loading it.

2. Check how you're loading the file in code #

If you’re manually loading the configuration file in your CrawlTopology class, make sure you’re using the correct classpath-based method. For example:

// Load from root classpath (note the leading slash)
InputStream filterStream = getClass().getResourceAsStream("/parsefilter.json");
if (filterStream == null) {
    throw new RuntimeException("Custom parsefilter.json file not found");
}

Avoid using relative paths without context—classpath resources are resolved relative to the class's package unless you use the leading slash to target the root.

3. Ensure Maven packages the resource file #

Sometimes Maven might exclude certain files during packaging. Double-check your pom.xml to make sure it includes JSON files from the resources directory:

<build>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
            <includes>
                <include>**/*.json</include>
            </includes>
        </resource>
    </resources>
</build>

After running mvn package, open your generated JAR with a zip tool and confirm parsefilter.json exists inside (usually in the root directory or the subfolder you specified).

4. Explicitly set the config file path in your topology #

StormCrawler's JSoupParserBolt uses a configuration key to locate the parse filter file. If you’re using a custom filename or path, make sure you set this in your topology config:

Config crawlerConf = CrawlConfiguration.create();
// Point to your custom file (path relative to classroot)
crawlerConf.put(JSoupParserBolt.PARSE_FILTER_CONFIG_FILE, "my-custom-parsefilter.json");

This overrides the default lookup and ensures the bolt looks for your specific file.

5. Load the file externally (optional) #

If you prefer not to package the config inside the JAR, you can pass the file path as a system property when running the JAR:

java -jar your-crawler.jar -Dparsefilter.path=/absolute/path/to/parsefilter.json

Then adjust your code to read this property:

String filterPath = System.getProperty("parsefilter.path", "parsefilter.json");
File filterFile = new File(filterPath);
// Load the file from the specified path

内容的提问来源于stack exchange，提问作者Abhishek Kumar