Christian Bühlmann

TL:DR you still need to check the sources.

This use of #ChatGPT seems to be more applicable for students having to write a paper quickly than for serious #research. Or maybe as support for literarture review, but the search time optimisation will be lost in checking the sources. An interface towards a research database, allowing an automatic check of the citations, building on tools such as the ones as developed by Simon Willison, could be an answer. An interface towards a research database, allowing an automatic check of the citations, building on tools such as the ones as developed by Simon Willison, could be an answer.

If you know how to properly prompt ChatGPT, it will give you sources. Here’s how.

1. Write a query and ask ChatGPT

To start, you need to ask ChatGPT something that needs sources or citations. I’ve found it’s better to ask a question with a longer answer so there’s more « meat » for ChatGPT to chew on.

Keep in mind that ChatGPT can’t provide any information after 2021 and requests about information pre-internet (say, for a paper on Ronald Reagan’s presidency) will have far fewer available sources.

Here’s an example of a prompt I wrote on a topic I worked on a lot when I was in grad school:

Describe the learning theories of cognitivism, behaviorism, and constructivism

2. Ask ChatGPT to provide sources

This is where a bit of prompt engineering comes in. A good starting point is with this query:

Please provide sources for the previous answer

I’ve found that this often provides offline sources, books, papers, etc. The problem with offline sources is you can’t check their veracity. But it’s a starting point. A better query is this:

Please provide URL sources

This specifically tells ChatGPT that you want clickable links to sources. You can also tweak this up by asking for a specific quantity of sources, although your mileage may vary in terms of how many you get back:

Please provide 10 URL sources

In our next step, we’ll see what we can do with these.

3. Attempt to verify/validate the provided sources

Keep in mind this golden rule about ChatGPT-provided sources: ChatGPT is more often wrong than right.

Across the many times I’ve asked ChatGPT for URL sources, roughly half are just plain bad links. Another 25% or more are links that go to topics completely or somewhat unrelated to the one you’re trying to source.

For example, I asked for sources on a backgrounder for the phrase « trust but verify, » generally attributed to 1980s US President Ronald Reagan. I got a lot of sources back, but most didn’t actually exist. I got some back that correctly took me to active pages on the Reagan Presidential Library site, but where the page topic had nothing to do with the phrase in question.