如何将含周期性文本与坐标的.txt文件加载为NumPy数组并仅读取坐标行？

Hey there! Let's break down how to handle this large file efficiently, since your current approaches either hog memory or create extra work with string storage. The key here is leaning into your file's fixed repeating structure (2 text lines → 22 coordinate lines) to avoid loading the entire file into memory at once.

First, Let's Address Your Existing Approaches #

• Your first method loads all coordinates into a list, which is a problem for huge files—memory usage will scale with the total number of coordinates, which could get out of hand.
• Your second method writes coordinates as strings to a new file, but this adds unnecessary I/O overhead and forces you to re-parse everything later, which is inefficient.

The Optimal Solution: Process the File in Fixed Blocks #

Since we know exactly how the file is structured, we can skip the text lines in batches and read coordinate blocks directly. This keeps memory usage low (especially if you don't need all coordinates in memory at once) and avoids line-by-line checks for "C" or "H".

Option 1: Load All Coordinates into a NumPy Array (With Controlled Memory)

If you do need all coordinates stored, this method reads in blocks to avoid redundant checks, and only stores numeric values (no wasted space):

import numpy as np

def load_coords_from_large_file(filename):
    all_coords = []
    with open(filename, 'r') as f:
        while True:
            # Skip the 2 header text lines
            for _ in range(2):
                line = f.readline()
                if not line:  # Hit end of file
                    return np.array(all_coords, dtype=np.float64)
            
            # Read the next 22 coordinate lines
            block = []
            for _ in range(22):
                line = f.readline()
                if not line:
                    break  # Handle incomplete final block
                # Split on any whitespace (more robust than split(" "))
                _, x, y, z = line.strip().split()
                block.append([float(x), float(y), float(z)])
            
            if block:
                all_coords.extend(block)
    
    return np.array(all_coords, dtype=np.float64)

Option 2: Process Coordinates in Blocks (Minimal Memory Usage)

If you don't need all coordinates stored at once (e.g., running calculations on each block sequentially), you can process each coordinate block on the fly without saving everything to memory. This keeps your memory footprint constant (only one 22-line block in memory at a time):

def process_coords_blockwise(filename):
    with open(filename, 'r') as f:
        while True:
            # Skip 2 text lines
            for _ in range(2):
                if not f.readline():
                    return  # Exit if end of file is reached
            
            # Process the 22 coordinate lines
            block_coords = []
            for _ in range(22):
                line = f.readline()
                if not line:
                    break
                _, x, y, z = line.strip().split()
                block_coords.append([float(x), float(y), float(z)])
            
            # Do your work with this block here!
            # Example: print stats, run simulations, write to a structured output file
            print(f"Processed block with {len(block_coords)} coordinates")

Key Improvements Over Your Original Code #

1. No redundant checks: Instead of checking every line for "C" or "H", we use the file's fixed structure to know exactly where coordinates are—this speeds up processing significantly.
2. Robust splitting: Using split() (without arguments) handles any number of spaces/tabs between values, which is more reliable than split(" ").
3. Memory control: The blockwise approach lets you choose whether to store all coordinates or process them incrementally, making it suitable for files of any size.

内容的提问来源于stack exchange，提问作者ira_s16