寻求可基于图像序列生成字幕的开源算法（用于无声电影初步研究）

Hey there! Since you're doing initial research with low accuracy requirements, here are some solid open-source projects and algorithms that can generate captions from image sequences (like silent film clips):

Pre-Trained Multi-Modal Models (Quick to Prototype) #

• BLIP-2: This is a powerful multi-modal model that works great for both single images and sequential frames. For silent films, you can sample frames every 2-3 seconds (to keep computation manageable) and feed them into the model. It’s built on PyTorch and has easy-to-use implementations via the transformers library—no heavy custom code needed. Since you don’t need perfect accuracy, the pre-trained weights will give you decent, contextually relevant captions right out of the box.
• VideoBERT: Designed specifically for video sequence understanding, this model captures temporal relationships between frames (like character movements or scene transitions) better than single-image models. It has open-source implementations for both TensorFlow and PyTorch. You can feed it either pre-extracted frames or even raw video files (some forks support direct video input) to generate coherent, sequence-aware captions.

Customizable Video Understanding Pipelines #

• SlowFast Networks + Captioning Head: SlowFast is a popular video recognition model from Meta that excels at capturing both slow, detailed actions and fast-paced movements. While it’s built for action recognition, you can easily add a captioning decoder (like an LSTM or Transformer) on top and fine-tune it with small video caption datasets (like MSVD or MSR-VTT). The codebase is well-documented, making it perfect for research where you might want to tweak the pipeline later.

Lightweight DIY Approach #

• Seq2Seq with CNN Feature Extraction: If you want to build something from scratch to understand the basics, this is a great option. Use a pre-trained CNN (like ResNet) to extract visual features from each frame, then feed those sequential features into an LSTM-based Seq2Seq model to generate captions. This approach is lightweight, runs on modest hardware, and is easy to modify—ideal for initial exploratory research.

• Use ffmpeg to extract frames from your silent film clips efficiently:

  ffmpeg -i your_silent_film.mp4 -r 1/2 frames/%04d.jpg
  

  This command pulls one frame every 2 seconds and saves them to a frames folder.
• Since accuracy isn’t a top priority, skip fine-tuning pre-trained models unless you want to tailor captions to specific silent film genres (like silent comedies or dramas).

内容的提问来源于stack exchange，提问作者yoav.aviram