Try making automated tests for the docs

[nelsonjchen]

Make some questions and copy pastes of questions/situations from Discord.

Evaluate how AI reads the document. Use cheap thinking models.

Evaluate with another prompt on if the answer or answer traces matches the expected answer.

---

Extremely experimental and curious.

[calvinpark]

Not sure if I follow what you're saying.

Are you trying to find out whether the questions in Discord can be answered by optskug/doc, and if it can, give them the link?

[nelsonjchen]

Sure! Things like that. This is just spitballing.

[calvinpark]

This is how far I got haha

➜  discord-bot git:(main) ✗ python3 chat.py
Enter your question:
Which cars are supported?
ASK
Answer:
The following cars are supported:

1. `commaai/master-ci` (requires `commaai/extract_secoc_key_btn`)
2. `alexandresato/personal3`
3. `sunnypilot/master-new`
4. `chrispypatt/sunnypilot`
5. `optskug/SiennaFP`

*Note: `optskug/SiennaFP` is the only fork with TSK lateral support. The rest of the forks only have longitudinal support.*
Confidence: 100%


Enter your question:
Which forks are supported?
ASK
Answer:
The following forks are supported:

- `alexandresato/personal3`
- `sunnypilot/master-new`
- `chrispypatt/sunnypilot`
- `optskug/SiennaFP`
Confidence: 100%


Enter your question:
Which cars are supported?
ASK
Answer:
The following cars are supported by openpilot:

 - 2021-2023 RAV4 Prime
 - 2021-2023 Sienna Hybrid
 - 2020-2022 Yaris Hybrid (EUDM/JDM/MXDM)
Confidence: 100.0%

Super confident lies right there.

[nelsonjchen]

That's why it's great to use these to see where comprehension may fail.

[calvinpark]

I even asked it to provide "whether you have enough information to answer in 'enough_info' field", yet it said it does