如何避免向Cplex/Gurobi加载大型矩阵时同时存在两份实例？

Great question—dealing with 5-10GB constraint matrices is a classic pain point for large-scale optimization workflows. The dual memory footprint (workspace + solver struct) you're seeing is totally avoidable with these targeted fixes:

• Build constraints incrementally (skip pre-creating full matrix A)
  Don't waste memory storing the entire constraint matrix in your workspace first. Instead, add constraints directly to the solver object one (or a batch) at a time. This way, your workspace only holds small chunks of data, never the full 5-10GB matrix.
  Example for Gurobi (Python):

  import gurobipy as gp
  model = gp.Model()
  
  # First create your variables
  x = model.addVars(1000000, name="x")  # Example: 1M variables
  
  # Stream constraints from your data source (e.g., disk, database)
  for constraint_batch in load_constraint_batches():
      # Each batch has (coeffs, rhs, sense) for a subset of constraints
      for coeffs, rhs, sense in constraint_batch:
          expr = gp.LinExpr(coeffs, x)
          model.addConstr(expr, sense, rhs)
  

  CPLEX has equivalent methods like addLinearConstraint() and addLinearConstraints() that let you pass individual or batches of constraints without building the full matrix first.

• Pass sparse data directly without retaining workspace copies
  Both solvers are optimized for sparse matrices, but the key is to avoid keeping a full sparse matrix in your workspace. Use memory-mapped files (mmap) or generators to stream sparse matrix entries (row indices, column indices, values) directly to the solver, then discard each batch immediately after passing it.
  For example, in C++ with CPLEX, you can read sparse matrix rows from a file one by one, populate the solver's constraint data structures, and then free the temporary row memory right away.

• Force immediate garbage collection after passing data
  Even if you delete your workspace's matrix variable, some languages (like Python) don't always free memory instantly. Manually trigger garbage collection to ensure the workspace's matrix memory is released before the solver finishes loading its copy.
  Example in Python:

  import gc
  
  # After passing matrix A to the solver
  del A
  gc.collect()
  

  Make sure there are no other references to A (like in helper variables or functions) for this to work. In compiled languages like C++, use explicit memory deallocation or smart pointers to release the matrix memory immediately after passing.

• Load directly from disk (skip in-memory matrix entirely)
  If you can export your problem to a standard format (MPS, LP, or the solver's native format), let the solver read directly from the file. This eliminates the need to store any version of the matrix in your workspace—all memory usage is isolated to the solver's internal structures.
  Example for Gurobi:

  model = gp.Model()
  model.read("large_problem.mps")  # Reads directly from disk
  

  Just make sure you generate the MPS/LP file in a streaming way (writing rows to disk as you generate them) so you don't load the full matrix into memory to create the file.

• Tweak solver memory optimization parameters
  Both CPLEX and Gurobi have built-in settings to reduce memory usage:

  • Gurobi: Enable PreCrush to eliminate redundant constraints early, set MemoryLimit to cap solver memory usage, or adjust Threads (fewer threads can reduce memory overhead from parallel processing).
  • CPLEX: Set memory.emphasis to HIGH to prioritize memory efficiency, enable preprocessing.reduce to cut down on constraint count, or use barrier.memory settings if using the barrier solver.

Combining these strategies—like incremental constraint building plus explicit garbage collection—should eliminate that dual-memory overlap entirely. The best approach depends on how you generate your constraint data (streamable vs. precomputed) and the programming language you're using.

内容的提问来源于stack exchange，提问作者Day Vid