如何让io.Reader的Read方法等待指定字节全部就绪后读取？

问题背景 #

我正在实现一个实时上传代理，需接收原始二进制输入数据并按250MB分块转发至外部服务器。目前使用Gin作为Web框架，go-resty作为处理外部上传的HTTP客户端。

当前实现代码如下：

func chunker(rc io.ReadCloser, bytes int64) ([]byte, int64, error) {
    buf := make([]byte, bytes)
    n, err := rc.Read(buf)
    if err != nil {
        return []byte(""), 0, err
    }
    return buf[:n], int64(n), nil
}

func Upload(c *gin.Context) {
    w := c.Writer
    req := c.Request
    reqBody := req.Body
    reqBodySize := int64(req.ContentLength)
    bytesPerChunk := int64(262144000) // 250 MB

    for {
        var noOfBytes int64 = bytesPerChunk
        if reqBodySize < bytesPerChunk {
            noOfBytes = reqBodySize
        }
        noOfBytes -= 1 // (io.ReadCloser).Read always returns an EOF error if we don't subtract one byte of the total (which is the normal & expected behavior of file uploads by the way)
        if noOfBytes <= 0 {
            break
        }

        bytes, size, err := chunker(reqBody, noOfBytes)
        if err != nil {
            fmt.Println("body EOF")
            break
        }
        reqBodySize -= size

        upload, _, err := external.Upload(bytes)
        if err != nil {
            w.WriteString("an error has occured")
            return
        }
    }

    reqBody.Close()
    w.WriteString("upload complete")
}

遇到的问题是，由于上传实时进行，rc.Read(buf)不会等待上传者发送完首个250MB就跳转至下一分块。根据io.Reader规范：

Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered. Even if Read returns n < len(p), it may use all of p as scratch space during the call. If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more.

需要实现让读取操作等待指定字节数就绪后再返回的功能。

解决方案 #

Go标准库的io.ReadFull函数可直接解决该问题——它会持续调用底层Reader的Read方法，直到填满指定缓冲区，或遇到错误/EOF。

修改后的分块读取函数 #

替换原chunker函数，用io.ReadFull确保读取到指定字节数：

func chunker(rc io.ReadCloser, bytes int64) ([]byte, int64, error) {
    buf := make([]byte, bytes)
    n, err := io.ReadFull(rc, buf)
    // 处理非预期EOF：已读取部分数据但未填满缓冲区，返回已读内容
    if err == io.ErrUnexpectedEOF {
        return buf[:n], int64(n), nil
    }
    if err != nil {
        return nil, 0, err
    }
    return buf, int64(n), nil
}

优化后的上传处理函数 #

移除原代码中noOfBytes -= 1的冗余处理，同时用defer确保请求体资源被正确释放：

func Upload(c *gin.Context) {
    w := c.Writer
    req := c.Request
    reqBody := req.Body
    defer reqBody.Close() // 确保函数结束时关闭请求体，避免资源泄漏
    reqBodySize := int64(req.ContentLength)
    bytesPerChunk := int64(262144000) // 250 MB

    for reqBodySize > 0 {
        chunkSize := bytesPerChunk
        if reqBodySize < bytesPerChunk {
            chunkSize = reqBodySize
        }

        bytes, size, err := chunker(reqBody, chunkSize)
        if err != nil {
            if err != io.EOF {
                w.WriteString("an error has occurred")
                return
            }
            break
        }
        reqBodySize -= size

        _, _, err := external.Upload(bytes)
        if err != nil {
            w.WriteString("an error has occurred")
            return
        }
    }

    w.WriteString("upload complete")
}

关键说明 #

• io.ReadFull会循环读取直到缓冲区被填满，若中途遇到EOF则返回io.ErrUnexpectedEOF，此时可正常返回已读取的部分数据。
• 用defer reqBody.Close()替代手动关闭，确保无论函数执行路径如何，资源都会被释放。
• 原代码中noOfBytes -=1是对io.Reader行为的错误妥协，io.ReadFull可正确处理完整分块的读取逻辑，无需该操作。

内容的提问来源于stack exchange，提问作者Mario