Android中基于ScrapingHub的定时爬虫数据通知系统实现需求

Got it, let's walk through building this Android notification system that checks your Scrapy project on Scrapinghub every 12 hours and triggers notifications based on new data. Here's a step-by-step, practical approach:

First, you'll need to pull your Scrapy project's output using Scrapinghub's API. Here's what to do:

• Grab your Scrapinghub API key from your account settings (keep this secure—don't hardcode it in your app; use Android's EncryptedSharedPreferences or a secure config service).
• Use the Items API to fetch the latest scraped data. For example, you can request items from your most recent finished job, or filter by a time range to avoid pulling old data repeatedly.

Example API Call with OkHttp #

val client = OkHttpClient()
val request = Request.Builder()
    .url("https://app.scrapinghub.com/api/items/<PROJECT_ID>/<JOB_ID>?apikey=<YOUR_API_KEY>")
    .build()

client.newCall(request).enqueue(object : Callback {
    override fun onFailure(call: Call, e: IOException) {
        // Handle network errors (log, retry later via WorkManager)
    }

    override fun onResponse(call: Call, response: Response) {
        response.body?.let { body ->
            val jsonData = body.string()
            // Parse JSON into your custom data model (use Gson/Moshi for easier parsing)
        }
    }
})

Replace <PROJECT_ID>, <JOB_ID>, and <YOUR_API_KEY> with your actual values. You can also use the Jobs API to dynamically fetch the latest completed job ID if your Scrapy jobs run on a recurring schedule.

For reliable background scheduling (even when the app is closed or the device is in Doze mode), use WorkManager—it's Google's recommended solution for deferrable, periodic tasks that need to survive device reboots.

Step 2.1: Add WorkManager Dependency #

Add this to your app-level build.gradle (Module level):

dependencies {
    implementation "androidx.work:work-runtime-ktx:2.8.1"
}

Step 2.2: Create a Worker Class #

This class will handle the API call, data comparison, and notification logic:

class ScrapingDataCheckWorker(
    context: Context,
    params: WorkerParameters
) : CoroutineWorker(context, params) {

    override suspend fun doWork(): Result {
        return try {
            // 1. Fetch latest data from Scrapinghub API
            val newData = fetchScrapinghubData()
            // 2. Compare with locally stored last processed data
            val hasNewItems = compareWithLocalData(newData)
            // 3. Trigger notification if new items exist
            if (hasNewItems) {
                showNewDataNotification(applicationContext, newData.size)
            }
            Result.success()
        } catch (e: Exception) {
            // Retry on failure with backoff (adjust retry policy as needed)
            Result.retry()
        }
    }

    // Implement fetchScrapinghubData() and compareWithLocalData() here
}

Step 2.3: Schedule the Periodic Task #

In your Application class or main activity, set up the 12-hour recurring task:

val periodicWorkRequest = PeriodicWorkRequestBuilder<ScrapingDataCheckWorker>(12, TimeUnit.HOURS)
    .setConstraints(Constraints.Builder()
        .setRequiredNetworkType(NetworkType.CONNECTED) // Only run when device is online
        .build())
    .build()

WorkManager.getInstance(context).enqueueUniquePeriodicWork(
    "ScrapingDataCheck",
    ExistingPeriodicWorkPolicy.KEEP, // Keep existing task if already scheduled
    periodicWorkRequest
)

To avoid spamming users with duplicate notifications, you need to store the last state of the scraped data locally.

3.1 Store Last Fetched Data #

Use Room Database (for structured, complex data) or EncryptedSharedPreferences (for simple checks like last item ID/timestamp) to track what you've already notified the user about. For example, if your scraped items have a unique ID, store the highest ID you've processed.

3.2 Trigger Notifications #

When you fetch new data and find unprocessed items, create a notification using NotificationManagerCompat:

private fun showNewDataNotification(context: Context, newItemCount: Int) {
    val notificationId = 1
    val channelId = "ScrapingUpdates"

    // Create notification channel (required for Android 8.0+)
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
        val channel = NotificationChannel(
            channelId,
            "Scraping Updates",
            NotificationManager.IMPORTANCE_DEFAULT
        ).apply {
            description = "Alerts for new data from your Scrapy project"
        }
        val notificationManager = context.getSystemService(NotificationManager::class.java)
        notificationManager.createNotificationChannel(channel)
    }

    val intent = Intent(context, MainActivity::class.java)
    val pendingIntent = PendingIntent.getActivity(
        context,
        0,
        intent,
        PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE
    )

    val notification = NotificationCompat.Builder(context, channelId)
        .setSmallIcon(R.drawable.ic_notification)
        .setContentTitle("New Scraped Data Available")
        .setContentText("Found $newItemCount new items from your project")
        .setPriority(NotificationCompat.PRIORITY_DEFAULT)
        .setContentIntent(pendingIntent)
        .setAutoCancel(true)
        .build()

    NotificationManagerCompat.from(context).notify(notificationId, notification)
}

• Network Errors: Add retry logic in your Worker (using Result.retry()) and set a backoff delay to avoid overwhelming the API.
• Android 13+ Notification Permissions: Request the POST_NOTIFICATIONS permission at runtime before showing notifications—users need to opt in.
• Data Parsing: Use a robust JSON parser like Moshi or Gson to handle unexpected data formats from the API.
• Battery Optimization: Inform users to exclude your app from battery optimization if they want 100% reliable 12-hour checks (WorkManager handles most cases, but strict battery saving modes can delay tasks).

内容的提问来源于stack exchange，提问作者Shalanki Gupta