Windows11下执行hdfs namenode -format遇URI语法错误求助

环境信息 #

• 操作系统：Windows 11
• JDK版本：jdk-11.0.16.1
• Hadoop版本：3.3.4

以管理员权限在Windows Terminal中执行命令 hdfs namenode -format 时，出现错误提示：ERROR common.Util: Syntax error in URI C:\hadoop-3.3.4\data\namenode，完整执行日志如下：

STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb; compiled by 'stevel' on 2022-07-29T12:32Z
STARTUP_MSG:   java = 11.0.16.1
************************************************************/
2022-12-26 20:17:07,811 INFO namenode.NameNode: createNameNode [-format]
2022-12-26 20:17:08,323 INFO common.Util: Assuming 'file' scheme for path /hadoop-3.3.4/data/namenode in configuration.
2022-12-26 20:17:08,323 INFO common.Util: Assuming 'file' scheme for path /hadoop-3.3.4/data/namenode in configuration.
2022-12-26 20:17:08,328 INFO namenode.NameNode: Formatting using clusterid: CID-eb3e0a34-9718-4003-afbb-508235763fef
2022-12-26 20:17:08,371 INFO namenode.FSEditLog: Edit logging is async:true
2022-12-26 20:17:08,404 INFO namenode.FSNamesystem: KeyProvider: null
2022-12-26 20:17:08,408 INFO namenode.FSNamesystem: fsLock is fair: true
2022-12-26 20:17:08,408 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2022-12-26 20:17:08,444 INFO namenode.FSNamesystem: fsOwner                = Fisnik (auth:SIMPLE)
2022-12-26 20:17:08,444 INFO namenode.FSNamesystem: supergroup             = supergroup
2022-12-26 20:17:08,447 INFO namenode.FSNamesystem: isPermissionEnabled    = true
2022-12-26 20:17:08,447 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2022-12-26 20:17:08,447 INFO namenode.FSNamesystem: HA Enabled: false
2022-12-26 20:17:08,504 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-12-26 20:17:08,511 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2022-12-26 20:17:08,511 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2022-12-26 20:17:08,511 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2022-12-26 20:17:08,515 INFO blockmanagement.BlockManager: The block deletion will start around 2022 Dec 26 20:17:08
2022-12-26 20:17:08,515 INFO util.GSet: Computing capacity for map BlocksMap
2022-12-26 20:17:08,517 INFO util.GSet: VM type       = 64-bit
2022-12-26 20:17:08,517 INFO util.GSet: 2.0% max memory 1000 MB = 20 MB
2022-12-26 20:17:08,517 INFO util.GSet: capacity      = 2^21 = 2097152 entries
2022-12-26 20:17:08,523 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2022-12-26 20:17:08,523 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManager: defaultReplication         = 1
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManager: maxReplication             = 512
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManager: minReplication             = 1
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2022-12-26 20:17:08,528 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2022-12-26 20:17:08,547 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2022-12-26 20:17:08,547 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2022-12-26 20:17:08,547 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2022-12-26 20:17:08,547 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2022-12-26 20:17:08,557 INFO util.GSet: Computing capacity for map INodeMap
2022-12-26 20:17:08,559 INFO util.GSet: VM type       = 64-bit
2022-12-26 20:17:08,559 INFO util.GSet: 1.0% max memory 1000 MB = 10 MB
2022-12-26 20:17:08,559 INFO util.GSet: capacity      = 2^20 = 1048576 entries
2022-12-26 20:17:08,559 INFO namenode.FSDirectory: ACLs enabled? true
2022-12-26 20:17:08,559 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2022-12-26 20:17:08,559 INFO namenode.FSDirectory: XAttrs enabled? true
2022-12-26 20:17:08,559 INFO namenode.NameNode: Caching file names occurring more than 10 times
2022-12-26 20:17:08,563 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2022-12-26 20:17:08,568 INFO snapshot.SnapshotManager: SkipList is disabled
2022-12-26 20:17:08,568 INFO util.GSet: Computing capacity for map cachedBlocks
2022-12-26 20:17:08,568 INFO util.GSet: VM type       = 64-bit
2022-12-26 20:17:08,571 INFO util.GSet: 0.25% max memory 1000 MB = 2.5 MB
2022-12-26 20:17:08,571 INFO util.GSet: capacity      = 2^18 = 262144 entries
2022-12-26 20:17:08,578 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2022-12-26 20:17:08,578 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2022-12-26 20:17:08,578 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2022-12-26 20:17:08,580 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2022-12-26 20:17:08,580 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2022-12-26 20:17:08,580 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2022-12-26 20:17:08,580 INFO util.GSet: VM type       = 64-bit
2022-12-26 20:17:08,584 INFO util.GSet: 0.029999999329447746% max memory 1000 MB = 307.2 KB
2022-12-26 20:17:08,584 INFO util.GSet: capacity      = 2^15 = 32768 entries
2022-12-26 20:17:08,608 INFO namenode.FSImage: Allocated new BlockPoolId: BP-164571709-192.168.38.107-1672082228600
2022-12-26 20:17:08,660 INFO common.Storage: Storage directory C:\hadoop-3.3.4\data\namenode has been successfully formatted.
2022-12-26 20:17:08,692 INFO namenode.FSImageFormatProtobuf: Saving image file C:\hadoop-3.3.4\data\namenode\current\fsimage.ckpt_0000000000000000000 using no compression
2022-12-26 20:17:08,772 INFO namenode.FSImageFormatProtobuf: Image file C:\hadoop-3.3.4\data\namenode\current\fsimage.ckpt_0000000000000000000 of size 401 bytes saved in 0 seconds .
2022-12-26 20:17:08,787 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2022-12-26 20:17:08,808 INFO namenode.FSNamesystem: Stopping services started for active state
2022-12-26 20:17:08,808 INFO namenode.FSNamesystem: Stopping services started for standby state
2022-12-26 20:17:08,812 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2022-12-26 20:17:08,812 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at root/192.168.38.107
************************************************************/

解决方法 #

1. 修正配置文件中的路径格式
   打开HADOOP_HOME/etc/hadoop/hdfs-site.xml，找到dfs.namenode.name.dir配置项，将路径修改为以下两种格式之一：

   • 使用正斜杠：<value>file:///C:/hadoop-3.3.4/data/namenode</value>
   • 使用双反斜杠：<value>file:///C:\\hadoop-3.3.4\\data\\namenode</value>
     必须加上file:///前缀，且路径分隔符不能用单反斜杠。

2. 检查环境变量配置
   确认HADOOP_HOME环境变量指向Hadoop的安装根目录（如C:\hadoop-3.3.4），且路径中没有空格、中文或特殊字符。

3. 确保目录权限与存在性

   • 手动创建C:\hadoop-3.3.4\data\namenode目录
   • 右键该目录，选择「属性」-「安全」，确保当前用户拥有完全控制权限

4. 清理残留文件后重新格式化
   如果之前执行过格式化，先删除namenode目录下的所有文件，再重新执行hdfs namenode -format命令。

内容的提问来源于stack exchange，提问作者Alix Blaine