部署BI Publisher的服务器每日固定时段挂起，求故障诊断方法

Hey there, let's walk through how to diagnose this daily scheduled hang issue with your BI Publisher + WebLogic setup. Since you're not super familiar with server operations, I'll keep this step-by-step and easy to follow:

Logs are your first and best clue—they’ll usually tell you what’s going wrong right before or during the hang. Here’s where to look:

• WebLogic Admin Server logs: Navigate to $DOMAIN_HOME/servers/AdminServer/logs (replace $DOMAIN_HOME with your actual WebLogic domain path). Focus on AdminServer.log and AdminServer.out. Look for errors, timeouts, OutOfMemoryError (OOM) entries, or messages about blocked threads around the hang time.
• BI Publisher logs: Check $DOMAIN_HOME/servers/<BIP_SERVER_NAME>/logs (use your BI Publisher server’s name here) for bipublisher.log. Look for any scheduled reports, data model refreshes, or export jobs running exactly when the hang starts—these are common culprits.
• Use the grep command to filter logs by time. For example, if the hang hits at 2 AM every day:

  grep "2024-XX-XX 02:" /path/to/AdminServer.log >> /tmp/hang_time_logs.txt
  

  Replace the date with the actual day you’re investigating.

Resource exhaustion (CPU, memory, disk) is a frequent cause of hangs. If you can access the server during the hang (or set up pre-emptive monitoring):

• Run top to see if the WebLogic Java process is hogging 100% CPU or using nearly all available memory.
• Check disk space with df -h—full disks (especially where logs or BI Publisher temp files are stored) can crash applications.
• Use iostat or vmstat to check for high disk I/O wait times—slow disk access can make the server unresponsive.
• If you can’t monitor in real-time, set up a cron job to log resources every minute around the hang window:

  */1 1-2 * * * top -b -n 1 >> /tmp/weblogic_resource_monitor.txt
  

  This will log CPU/memory usage from 1 AM to 2 AM daily.

Since the hang happens at a fixed time every day, a scheduled task is almost certainly triggering it. Check these places:

• WebLogic Console (once restored): Go to Deployments > BI Publisher and look for scheduled reports, data model refreshes, or batch export jobs running at the hang time.
• WebLogic Internal Schedulers: In the console, navigate to Environment > Servers > AdminServer > Configuration > Startup & Shutdown to check for any system-level timed tasks.
• OS-Level Cron Jobs: Run crontab -l (on Linux) to see if any scripts (like backups, data syncs, or maintenance) are running at the same time—these could be competing for resources with WebLogic.

If logs don’t give you a clear answer, thread and heap dumps will show you exactly what’s blocking the server:

• First, find the WebLogic Java process ID with ps aux | grep weblogic.
• Generate a thread dump (shows which threads are stuck):

  jstack <PROCESS_ID> > /tmp/weblogic_thread_dump.txt
  

• Generate a heap dump (if memory issues are suspected):

  jmap -dump:format=b,file=/tmp/weblogic_heap_dump.hprof <PROCESS_ID>
  

• You can analyze these files with tools like VisualVM (free) or Thread Dump Analyzer—look for deadlocked threads, or large objects (like unprocessed BI Publisher reports) taking up memory.

BI Publisher-specific settings might be causing the hang:

• Check Data Sources: If your reports connect to a database, make sure the database isn’t running maintenance (backups, index rebuilds) at the hang time—slow database queries can freeze BI Publisher, which then blocks WebLogic.
• Report Schedules: Are multiple large reports running at the same time? Too many concurrent reports can overwhelm the server’s resources.
• Temp File Cleanup: BI Publisher generates temp files for reports—check if these are being cleaned up automatically. If not, a full temp directory can cause hangs.

If you’re still stuck, try isolating variables to narrow down the cause:

• Temporarily disable all BI Publisher scheduled jobs. If the server stops hanging, re-enable them one by one to find the problematic job.
• Turn off any OS-level cron jobs running at the hang time to rule out external interference.

内容的提问来源于stack exchange，提问作者Benjamin