如何用Python高效搜索竖线分隔大文本文件首列的目标手机号

Hey there! Let's fix this slow search issue and make sure we're only matching the first column as you need.

The Problem With Your Original Code #

Your current code checks if the phone number exists anywhere in the line, which not only risks false matches (if the number shows up in another column) but also wastes time scanning the entire line for every row. Plus, you’re looping through all 10 million rows even if you find the match early—no need for that if the phone number is unique!

Solution 1: Optimized Single Query #

We’ll modify the code to only check the first column (split the line by | once to grab the phone number) and stop searching as soon as we find a match (if you know the number is unique). Here's the improved code:

target_phone = '999995555'

# Use `with` to auto-close the file (safer than manual close operations)
with open("F:/.../master.txt", "rt") as f:
    for line in f:
        # Split only once to get the first column, strip extra whitespace
        first_col = line.split('|', 1)[0].strip()
        if first_col == target_phone:
            print(line.strip())  # Remove messy newlines from output
            # If you’re sure there’s only one matching row, break to stop early!
            break

Why This Is Faster:

• Precise matching: We only compare the first column, so no false positives from numbers in other columns.
• Faster check: Comparing two short strings (the target phone and first column) is way quicker than scanning the entire line with in.
• Early exit: Using break stops the loop immediately after finding the match, saving time on the remaining rows.

Solution 2: For Multiple Queries (Even Faster!) #

If you need to search for multiple phone numbers, it’s worth building an indexed lookup first. Using a lightweight SQLite database will let you query in milliseconds after an initial import:

import sqlite3

# Create an in-memory database (use a file path like "phone_db.sqlite" to save it permanently)
conn = sqlite3.connect(':memory:')
cursor = conn.cursor()

# Create a table with an index on the phone number (for lightning-fast lookups)
cursor.execute('CREATE TABLE phone_records (phone TEXT PRIMARY KEY, full_line TEXT)')

# Import the data once (this takes time upfront but pays off for repeated queries)
with open("F:/.../master.txt", "rt") as f:
    for line in f:
        # Split to get the first column and the rest of the line
        phone, full_line = line.split('|', 1)
        phone = phone.strip()
        # Insert, ignoring duplicates if the same phone appears multiple times
        cursor.execute('INSERT OR IGNORE INTO phone_records VALUES (?, ?)', (phone, line.strip()))

conn.commit()

# Now query any phone number in seconds (or less!)
target_phone = '999995555'
cursor.execute('SELECT full_line FROM phone_records WHERE phone = ?', (target_phone,))
result = cursor.fetchone()

if result:
    print(result[0])
else:
    print("Phone number not found.")

conn.close()

Why This Works:

• SQLite indexes the phone number column, turning lookups from O(n) to O(1) time.
• Once imported, you can run as many queries as you want without reprocessing the entire 5GB file.

内容的提问来源于stack exchange，提问作者Rajesh K. Pandey