如何获取特定地区GitHub仓库PR及语言时间趋势并实现可视化

Hey there! Let's tackle your two GitHub data needs one by one—both are totally achievable with the right tools and APIs. Here's a step-by-step breakdown:

The core tool here is the GitHub Search API, which lets you filter PRs by repository language, owner location, and more. Here's how to make it work:

Step 1: Use the GitHub Search API for PRs #

You'll call the /search/issues endpoint (since PRs are categorized as issues in GitHub's API) with a targeted query string. For example, to find all PRs in Python repositories owned by users in the UK, your query would look like:

is:pr language:python user:location:"United Kingdom"

• is:pr: Filters results to only pull requests
• language:python: Targets repositories tagged with Python
• user:location:"United Kingdom": Narrows down to repos owned by users who listed the UK as their location

You can add extra filters like is:open/is:closed to target PR status, or created:>=2023-01-01 to focus on recent PRs.

Step 2: Handle Pagination & Rate Limits #

GitHub's API restricts unauthenticated requests to 10 per minute, while authenticated requests get 30 per minute. To fetch complete results:

• Use a GitHub personal access token (PAT) in your request headers for higher limits: Authorization: token YOUR_PAT
• Follow the Link header in API responses to paginate through results (look for the rel="next" link)

Step 3: Example Python Script #

Here's a quick script to fetch PRs using the requests library:

import requests

GITHUB_PAT = "your-personal-access-token"
headers = {"Authorization": f"token {GITHUB_PAT}"}

def fetch_region_lang_prs(location, language, page=1):
    url = "https://api.github.com/search/issues"
    query = f"is:pr language:{language} user:location:\"{location}\""
    params = {"q": query, "page": page, "per_page": 100}  # Max 100 results per page
    
    response = requests.get(url, headers=headers, params=params)
    response.raise_for_status()  # Raise error for HTTP issues
    return response.json()

# Fetch first page of UK Python PRs
uk_python_prs = fetch_region_lang_prs("United Kingdom", "Python")
print(f"Total PRs found: {uk_python_prs['total_count']}")

To replicate Githut's style with a region filter, you'll need three key steps: data collection, data processing, and interactive visualization.

Step 1: Collect Historical Repository Data #

The GitHub Search API only returns up to 1000 results, which isn't enough for long-term trends. For full historical data, use GitHub's BigQuery Public Dataset—it includes all public repository data dating back to 2011.

Here's a sample SQL query to get UK repository language counts by quarter:

SELECT
  EXTRACT(YEAR FROM created_at) AS year,
  EXTRACT(QUARTER FROM created_at) AS quarter,
  language,
  COUNT(*) AS repo_count
FROM
  `bigquery-public-data.github_repos.repos`
WHERE
  -- Handle varied location entries (UK, United Kingdom, London, etc.)
  REGEXP_CONTAINS(location, r"United Kingdom|UK|England|Scotland|Wales|Northern Ireland")
  AND language IS NOT NULL
GROUP BY
  year, quarter, language
ORDER BY
  year, quarter, repo_count DESC

Step 2: Process the Data #

Export the BigQuery results to a CSV or JSON file, then clean and transform it to calculate language percentages per time period:

• Calculate the total number of repos per quarter
• For each language, compute (repo_count / total_repos) * 100 to get its percentage share

Step 3: Build the Interactive Visualization #

Githut uses D3.js for its interactive charts, so we'll follow that approach. Here's a simplified framework to add a region filter and time-based navigation:

HTML Setup

<div>
  <select id="region-select">
    <option value="uk">United Kingdom</option>
    <!-- Add other regions here -->
  </select>
  <select id="time-select"></select>
</div>
<div id="chart"></div>

D3.js Code

// Load your processed data (replace with your file path)
d3.json("uk-language-trends.json").then(data => {
  // Group data by time period (year-quarter)
  const timeGroups = d3.group(data, d => `${d.year}-Q${d.quarter}`);
  const timePeriods = Array.from(timeGroups.keys()).sort();

  // Populate time selector
  d3.select("#time-select")
    .selectAll("option")
    .data(timePeriods)
    .join("option")
    .attr("value", d => d)
    .text(d => d);

  // Initialize chart with the first time period
  drawChart(timeGroups.get(timePeriods[0]));

  // Update chart when time/region changes
  d3.select("#time-select").on("change", function() {
    drawChart(timeGroups.get(this.value));
  });

  // Core chart drawing function
  function drawChart(periodData) {
    const margin = {top: 20, right: 30, bottom: 70, left: 60};
    const width = 960 - margin.left - margin.right;
    const height = 500 - margin.top - margin.bottom;

    // Clear previous chart
    d3.select("#chart").html("");

    const svg = d3.select("#chart")
      .append("svg")
      .attr("width", width + margin.left + margin.right)
      .attr("height", height + margin.top + margin.bottom)
      .append("g")
      .attr("transform", `translate(${margin.left},${margin.top})`);

    // Calculate percentages
    const totalRepos = d3.sum(periodData, d => d.repo_count);
    periodData.forEach(d => d.percentage = (d.repo_count / totalRepos) * 100);

    // Sort languages by percentage
    periodData.sort((a, b) => b.percentage - a.percentage);

    // Scales
    const x = d3.scaleBand()
      .domain(periodData.map(d => d.language))
      .range([0, width])
      .padding(0.1);

    const y = d3.scaleLinear()
      .domain([0, 100])
      .range([height, 0]);

    // Draw bars
    svg.selectAll(".bar")
      .data(periodData)
      .join("rect")
      .attr("class", "bar")
      .attr("x", d => x(d.language))
      .attr("y", d => y(d.percentage))
      .attr("width", x.bandwidth())
      .attr("height", d => height - y(d.percentage))
      .attr("fill", d => getLanguageColor(d.language)) // Add your own color mapping
      .on("mouseover", function(event, d) {
        // Add hover tooltip here
        d3.select(this).attr("opacity", 0.7);
      })
      .on("mouseout", function(event, d) {
        d3.select(this).attr("opacity", 1);
      });

    // Axes
    svg.append("g")
      .attr("transform", `translate(0,${height})`)
      .call(d3.axisBottom(x))
      .selectAll("text")
      .attr("transform", "rotate(-45)")
      .style("text-anchor", "end");

    svg.append("g")
      .call(d3.axisLeft(y).ticks(10).tickFormat(d => `${d}%`));

    // Y-axis label
    svg.append("text")
      .attr("transform", "rotate(-90)")
      .attr("y", -margin.left + 20)
      .attr("x", -height/2)
      .style("text-anchor", "middle")
      .text("Percentage of Repositories");
  }

  // Helper function to assign consistent colors to languages
  function getLanguageColor(language) {
    const colorMap = {
      "Python": "#3776AB",
      "JavaScript": "#F7DF1E",
      "Java": "#007396",
      // Add more languages here
    };
    return colorMap[language] || "#999";
  }
});

Key Notes #

• For region switching, pre-process data for multiple regions and update the chart when the region selector changes.
• If you prefer a no-code/low-code approach, tools like Tableau or Plotly Express can generate similar interactive charts without writing raw D3.js.

内容的提问来源于stack exchange，提问作者snow_fall