Drawing graphs with aider, GPT-4o and matplotlib

I recently wanted to draw a graph showing how LLM code editing skill has been changing over time as new models have been released by OpenAI, Anthropic and others. I have all the data in a YAML file that is used to render aider’s LLM leaderboards.

Below is the aider chat transcript, which shows:

  • I launch aider with the YAML file, a file with other plots I’ve done recently (so GPT can crib the style) and an empty file called over_time.py.
  • Then I ask GPT to draw the scatterplot I want.
  • I run the resulting script and share the error output with GPT so it can fix a small bug.
  • I ask it to color the points for GPT-4 and GPT-3.5 family models differently, to better see trends within those model families.
  • I add a PNG of the graph to the chat, so GPT can see the current graph and understand why I want it to move the point labels so they don’t overlap the points.
  • I work through a series of other small style changes, like changing fonts and the graph border.

In the end I have the graph, but I also have the python code in my repo. So I can update this graph easily whenever I add new entries to the YAML data file.

Aider chat transcript

Aider v0.35.1-dev

Models: gpt-4o with diff edit format, weak model gpt-3.5-turbo

Added _data/edit_leaderboard.yml to the chat.

Added benchmark/over_time.py to the chat.

Added benchmark/plots.py to the chat.

in over_time.py, draw a graph for each of the yaml entries with a released date. plot date on x, pass_rate_2 on y and label with the model name. do a scatter plot. use colors/styles similar to the other graphs.

To draw a scatter plot for each of the YAML entries with a released date, we need to:

  1. Read the YAML file.
  2. Extract the released date and pass_rate_2 for each entry.
  3. Plot the data using matplotlib.

Here are the changes:

benchmark/over_time.py

import matplotlib.pyplot as plt
import yaml
from datetime import datetime

def plot_over_time(yaml_file):
    with open(yaml_file, 'r') as file:
        data = yaml.safe_load(file)

dates = []
    pass_rates = []
    models = []

for entry in data:
        if 'released' in entry and 'pass_rate_2' in entry:
            dates.append(entry['released'])
            pass_rates.append(entry['pass_rate_2'])
            models.append(entry['model'])

plt.figure(figsize=(10, 6))
    colors = ['red' if 'gpt-4' in model else 'green' if 'gpt-3.5' in model else 'blue' for model in models]
    plt.scatter(dates, pass_rates, c=colors, alpha=0.5, s=100)

for i, model in enumerate(models):
        plt.annotate(model, (dates[i], pass_rates[i]), fontsize=8, alpha=0.75,
                    xytext=(5, 5), textcoords='offset points')

plt.xlabel('Release Date')
    plt.ylabel('Pass Rate 2')
    plt.title('Model Performance Over Time')
    for spine in ax.spines.values():
        spine.set_edgecolor('#DDDDDD')
        spine.set_linewidth(0.5)
    plt.tight_layout()
    plt.savefig("tmp_over_time.svg")
    imgcat(fig)

# Example usage
plot_over_time('_data/edit_leaderboard.yml')

Applied edit to benchmark/over_time.py

Commit a018fdf aider: Make the dots bigger.

Commit 85d9d84 aider: Color the “gpt-3.5” models green in the plotting code.

Commit 0d11357 aider: Color the “gpt-4” models differently.