从Google Storage读取Parquet到ClickHouse时出现超时读取错误求助

问题背景 #

有一个SQL脚本，通过S3函数从Google Cloud Storage读取Parquet文件写入ClickHouse表，该脚本正常运行约两年，今日突发异常。

报错信息 #

DBeaver执行查询时的错误： #

SQL Error [23] [07000]: Code: 23. DB::Exception: Cannot read from istream at offset 0: While executing ParquetBlockInputFormat: While executing S3. (CANNOT_READ_FROM_ISTREAM) (version 23.7.3.14 (official build))
, server ClickHouseNode [uri=http://servername.ru:8123/default, options={socket_timeout=30000000,use_server_time_zone=false,use_time_zone=false}]@1364421048

ClickHouse system.text_log中对应query_id的日志： #

Row 1:
──────
event_date:              2024-06-13
event_time:              2024-06-13 10:53:01
event_time_microseconds: 2024-06-13 10:53:01.291989
microseconds:            291989
thread_name:             VFSRead
thread_id:               45246
level:                   Error
query_id:                48eeab01-34f6-47d2-a452-901af146bb8e
logger_name:             AWSClient
message:                 Failed to make request to: ***path_to_google_storgae*** 000000000060events_20240612_1_of_6.parquet: Poco::Exception. Code: 1000, e.code() = 0, Timeout, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SecureSocketImpl::mustRetry(int, Poco::Timespan&) @ 0x0000000018250b4c in /usr/bin/clickhouse
1. Poco::Net::SecureSocketImpl::receiveBytes(void*, int, int) @ 0x0000000018251df4 in /usr/bin/clickhouse
2. Poco::Net::HTTPSession::refill() @ 0x000000001827246f in /usr/bin/clickhouse
3. Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long) @ 0x000000001826b680 in /usr/bin/clickhouse
4. ? @ 0x0000000018170668 in /usr/bin/clickhouse
5. std::basic_streambuf<char, std::char_traits<char>>::uflow() @ 0x0000000008f7f3ea in /usr/bin/clickhouse
6. std::basic_istream<char, std::char_traits<char>>::get() @ 0x0000000008f80a19 in /usr/bin/clickhouse
7. Poco::Net::HTTPResponse::read(std::basic_istream<char, std::char_traits<char>>&) @ 0x00000000182707ef in /usr/bin/clickhouse
8. Poco::Net::HTTPClientSession::receiveResponse(Poco::Net::HTTPResponse&) @ 0x000000001825e655 in /usr/bin/clickhouse
9. ? @ 0x0000000012e32685 in /usr/bin/clickhouse
10. DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const @ 0x0000000012e29192 in /usr/bin/clickhouse
11. DB::S3::PocoHTTPClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const @ 0x0000000012e28f7e in /usr/bin/clickhouse
12. Aws::Client::AWSClient::AttemptOneRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const @ 0x0000000018483b64 in /usr/bin/clickhouse
13. Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const @ 0x0000000018480902 in /usr/bin/clickhouse
14. Aws::Client::AWSClient::MakeRequestWithUnparsedResponse(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const @ 0x00000000184882e6 in /usr/bin/clickhouse
15. Aws::S3::S3Client::GetObject(Aws::S3::Model::GetObjectRequest const&) const @ 0x000000001855f919 in /usr/bin/clickhouse
16. DB::S3::Client::GetObject(DB::S3::ExtendedRequest<Aws::S3::Model::GetObjectRequest> const&) const @ 0x0000000012e06e7b in /usr/bin/clickhouse
17. DB::ReadBufferFromS3::sendRequest(unsigned long, std::optional<unsigned long>) const @ 0x0000000012e4ff03 in /usr/bin/clickhouse
18. DB::ReadBufferFromS3::initialize() @ 0x0000000012e4de4c in /usr/bin/clickhouse
19. DB::ReadBufferFromS3::nextImpl() @ 0x0000000012e4d579 in /usr/bin/clickhouse
20. DB::ReadBufferFromRemoteFSGather::readImpl() @ 0x0000000012eb6240 in /usr/bin/clickhouse
21. DB::ReadBufferFromRemoteFSGather::readInto(char*, unsigned long, unsigned long, unsigned long) @ 0x0000000012eb59f3 in /usr/bin/clickhouse
22. ? @ 0x0000000012de83e2 in /usr/bin/clickhouse
23. ? @ 0x0000000012de8b23 in /usr/bin/clickhouse
24. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000ea0b70f in /usr/bin/clickhouse
25. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000ea0e45c in /usr/bin/clickhouse
26. ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000ea0704f in /usr/bin/clickhouse
27. ? @ 0x000000000ea0d041 in /usr/bin/clickhouse
28. start_thread @ 0x0000000000008105 in /usr/lib64/libpthread-2.17.so
29. clone @ 0x00000000000feb2d in /usr/lib64/libc-2.17.so
 (version 23.7.3.14 (official build))
revision:                54476
source_file:             src/Common/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)
source_line:             225
message_format_string:   

环境对比与已尝试操作 #

• 拥有独立DEV环境，配置、SQL脚本及读取的Parquet文件与生产环境完全一致，但DEV环境中查询可正常执行
• 生产环境已尝试重启ClickHouse、多次执行查询，问题仍持续；DEV环境每次执行均无异常

内容的提问来源于stack exchange，提问作者Alexandr