重构Azure Data Lake Gen2文件上传至Express后端遇400错误

问题描述 #

原本可正常运行的React/TypeScript前端代码，能直接上传文件到Azure Data Lake Gen2。重构到Express后端后，在flush步骤触发400错误：

400 The uploaded data is not contiguous or the position query parameter value is not equal to the length of the file after appending the uploaded data.

测试文件大小为805B，position参数已设为对应值，怀疑文件传输到后端后格式被篡改，相关代码如下：

原前端正常代码 #

const onHandleChange = (event: any) => {
  const file: File = event.currentTarget.files[0];
  instance
    .acquireTokenSilent({
      scopes: ['https://storage.azure.com/.default'],
      account: accounts[0] as AccountInfo,
    })
    .then((response) => {
      const header = new Headers();
      header.append('Authorization', `Bearer ${response.accessToken}`);
      const path = `https://${ACCOUNT}.dfs.core.windows.net/${FILESYSTEM}/${USERNAME}/${FILENAME}`;

      fetch(`${path}?resource=file`, {
        method: 'PUT',
        headers: header,
      })
        .then(() =>
          fetch(`${path}?action=append&position=0`, {
            method: 'PATCH',
            headers: header,
            body: file,
          })
        )
        .then(() =>
          fetch(`${path}?action=flush&position=${file.size}`, {
            method: 'PATCH',
            headers: header,
            body: '',
          })
        )
        .then((resp) => {
          console.log(resp);
        })
        .catch((error) => {
          console.log(error);
        });
    })
    .catch((err) => console.log(err));
};

return (
  <div className="App">
    <p>Demo App</p>
    {isAuthenticated && (
      <form>
        <input type="file" id="fileField" onChange={onHandleChange}></input>
      </form>
    )}
    <UnauthenticatedTemplate>
      <button onClick={() => signInClickHandler(instance)}>Sign in</button>
    </UnauthenticatedTemplate>
  </div>
);

重构后前端代码 #

const onHandleChange = (event: any) => {
  const file: File = event.currentTarget.files[0];
  instance
    .acquireTokenSilent({
      scopes: ['https://storage.azure.com/.default'],
      account: accounts[0] as AccountInfo,
    })
    .then((response) => {
      const data = await file.arrayBuffer();
      fetch("/file", {
        method: "POST",
        headers: {
          Authorization: response.accessToken,
          "Content-Type": "application/x-www-form-urlencoded",
        },
        body: data,
      }).then((uploadResponse: Response | Promise<Response>) =>
        console.log(uploadResponse)
      );
    });
};

return (
  <div className="App">
    <p>Demo App</p>
    {isAuthenticated && (
      <form>
        <input type="file" id="fileField" onChange={onHandleChange}></input>
      </form>
    )}
    <UnauthenticatedTemplate>
      <button onClick={() => signInClickHandler(instance)}>Sign in</button>
    </UnauthenticatedTemplate>
  </div>
);

Express后端代码 #

router.post('/', function (req, res, next) {
  const accessToken = req.headers.authorization;
  const file = req.body;
  const size = req.headers['content-length'];
  const header = new fetch.Headers();
  header.append('Authorization', `Bearer ${accessToken}`);
  const path = `https://${ACCOUNT}.dfs.core.windows.net/${FILESYSTEM}/${USERNAME}/${FILENAME}`;

  fetch(`${path}?resource=file`, {
    method: 'PUT',
    headers: header,
  })
    .then(() =>
      fetch(`${path}?action=append&position=0`, {
        method: 'PATCH',
        headers: header,
        body: { ...file },
      })
    )
    .then(() =>
      fetch(`${path}?action=flush&position=${size}`, {
        method: 'PATCH',
        headers: header,
        body: '',
      })
    )
    .then((resp) => {
      console.log(resp);
    })
    .catch((error) => console.log(error));
});

问题根源及修复方案 #

1. 前端请求头错误 #

设置了Content-Type: application/x-www-form-urlencoded，但二进制文件不能用这个类型，会导致后端解析时破坏原始数据。移除该头或改为application/octet-stream。

2. 后端未配置二进制解析 #

Express默认不处理原始二进制请求体，需添加express.raw()中间件，否则req.body会是错误格式。

3. 后端请求体传递错误 #

body: { ...file }会把二进制数据转成对象，破坏原始内容，直接传递req.body即可。

4. 前端异步语法错误 #

then回调中使用await但未加async，导致语法错误。

修复后代码 #

修复后前端代码 #

const onHandleChange = (event: any) => {
  const file: File = event.currentTarget.files[0];
  instance
    .acquireTokenSilent({
      scopes: ['https://storage.azure.com/.default'],
      account: accounts[0] as AccountInfo,
    })
    .then(async (response) => {
      const data = await file.arrayBuffer();
      fetch("/file", {
        method: "POST",
        headers: {
          Authorization: response.accessToken,
          "Content-Type": "application/octet-stream",
        },
        body: data,
      }).then((uploadResponse: Response) =>
        console.log(uploadResponse)
      );
    })
    .catch((err) => console.log(err));
};

return (
  <div className="App">
    <p>Demo App</p>
    {isAuthenticated && (
      <form>
        <input type="file" id="fileField" onChange={onHandleChange}></input>
      </form>
    )}
    <UnauthenticatedTemplate>
      <button onClick={() => signInClickHandler(instance)}>Sign in</button>
    </UnauthenticatedTemplate>
  </div>
);

修复后Express后端代码 #

// 在路由注册前添加中间件，处理二进制请求体
app.use('/file', express.raw({ type: 'application/octet-stream', limit: '10mb' }));

router.post('/', function (req, res, next) {
  const accessToken = req.headers.authorization;
  const file = req.body; // 现在是原始Buffer
  const size = req.headers['content-length'];
  const header = new fetch.Headers();
  header.append('Authorization', `Bearer ${accessToken}`);
  const path = `https://${ACCOUNT}.dfs.core.windows.net/${FILESYSTEM}/${USERNAME}/${FILENAME}`;

  fetch(`${path}?resource=file`, {
    method: 'PUT',
    headers: header,
  })
    .then(() =>
      fetch(`${path}?action=append&position=0`, {
        method: 'PATCH',
        headers: header,
        body: file, // 直接传原始Buffer
      })
    )
    .then(() =>
      fetch(`${path}?action=flush&position=${size}`, {
        method: 'PATCH',
        headers: header,
        body: '',
      })
    )
    .then((resp) => {
      console.log(resp);
      res.status(resp.status).send(resp.statusText); // 给前端返回响应
    })
    .catch((error) => {
      console.log(error);
      res.status(500).send(error.message);
    });
});

内容的提问来源于stack exchange，提问作者savreline