A couple of years ago, I got into a discussion about Artificial Intelligence (AI) with a couple of people who write computer code (in Python) as part of their jobs. They suggested that I could use it to kick-start rants on this blog. At the time, ChatGPT was the AI version in the headlines, and they suggested that I should use it. Initially, I wanted to find out how I could get it to work for me, so I queried it about its capabilities. The ‘query’ I sent to it was: “I am just trying to obtain an understanding of the capabilities of ChatGPT and to work out if it will be of use to me”. Its responses to this and subsequent questions can be found in the ‘Having a Chat’ item I compiled1. As I related at the time, the experience was disturbing at the time for a person who had grown up without a computer anywhere except at the nearest university.
The second interaction with Chat GPT was to ask it where the myth about opals being bad luck had come from (I was editing a paper on Opals at the time). However, ChatGPT only reiterated the old legend that it had come from one of Sir Walter Scott’s novels, which subsequent research showed to be equine ordure. It was a disappointing interaction with ChatGPT2.
Some time later, I had cause to write a geological report on the Wiso Basin, a sedimentary basin in the western Northern Territory, and I thought ChatGPT would be able to compile the bare bones of an introduction which I could edit into something acceptable. It was disappointing to say the least. The result it gave me was completely incorrect, and is something I subsequently found out to be what those in the AI game call an ‘hallucination’ (i.e. complete bullshit)3. So I wrote my report without this dubious assistance.
A few days ago, I had a conversation with one of the coders mentioned above, and he told me that he uses DeepSeek, the AI ‘chatbot’ made in China. Its most recent version DeepSeek R1 is said to be as good or better than ChatGPT, while costing far less to create. While it can answer any questions, it has been ’trained’ to avoid politically sensitive questions (for China). For instance, when asked what happened in Tiananmen Square on the 4th of June 1989, it gave no details about the massacre4.
My coder suggested I send my Wiso Basin query to him and he would run it through several AI chatbots to see what came out and how it compared to my previous experience with ChatGPT. So, I sent him the query: “Explain the stratigraphy of the Northern Territory’s Wiso Basin.” Some bits of the result were hilarious. For instance, DeepSeek 8b llama, said “So first maybe I should figure out exactly where it is. It’s probably near some other regions I know, like Kakadu or Darwin. But wait, I think it’s more towards the southeast part of the NT, near the South China Sea. Maybe it’s part of the larger sedimentary basin”. You have to laugh. The Wiso Basin is in the central western Northern Territory. However, it gets worse. It suggests that the oldest sedimentary rocks were deposited in the basin during the Cretaceous. They weren’t. The oldest rock in the basin are early Cambrian in age, some 400+ million years older than the Cretaceous. They also stated that the sediments were deposited by “rivers and winds”. They weren’t; the rocks were deposited in a marine or marginal marine environment.
The answer from DeepSeek 8b llama was rubbish, so my coder tried the same query on the more powerful DeepSeek 32b qwen. This at least started off on the right foot, stating that the basement rocks (the rocks underlying the basin) were Precambrian and were about 1 billion years old. However, it got worse when it said that there are Paleozoic rocks, including Cambrian to Permian rocks. This was stated to be overlain by Triassic to Cretaceous rocks. This is rubbish too. As stated above, the Wiso Basin comprises mostly rocks of early Cambrian to Ordovician age, and these are overlain by Devonian sediments.
Then my coder tried using Grok v3, Elon Musk’s AI chatbot. This was partly better than DeepSeek in that it got the ages of the sediments roughly correct. It stated that the rocks in the basin extend from the middle Cambrian [they are early Cambrian in age] to Ordovician in age [it missed the Devonian Lake Surprise Sandstone]. It also lists some of the rock unit names (Montejinni Limestone, Hooker Creek Formation, Lothari Hill Sandstone], but misses the Hanson River beds and the Point Wakefield beds.
Next, my coder tried ChatGPT 4o (the pay for use version). It turned out to be quite well formatted but the content was just as ropey as the others. It correctly noted the Hooker Creek Formation and the Montejinni Limestone as being Cambrian, but included the Wiso Formation, which does not exist in the Australian Stratigraphic Units Database5. It does, however exist in a public report from a company involved in exploration in the area, but there are no details for that unit6.
It also states that there are Ordovician and Silurian sediments in the basin. While Ordovician rocks are present there are none from the Silurian. Similarly, it mentions the Devonian and Carboniferous. While there are sediments of Devonian age, there are none from the Carboniferous.
So, these chatbots have a long way to go before you can trust anything they say about the geology of the Wiso Basin, or indeed, probably anywhere. So, I keep plugging on, writing stuff myself.
Sources
- https://blotreport.com/2023/04/10/having-a-chat/
- https://blotreport.com/2023/04/15/the-opal-myth/
- https://blotreport.com/2023/04/23/dead-wrong/
- https://www.bbc.com/news/articles/c5yv5976z9po
- https://asud.ga.gov.au/search-stratigraphic-units
- https://www.proactiveinvestors.co.uk/companies/news/12983/western-desert-resources-planning-for-field-season-at-roper-bar-project-15873.html
In my experience Google’s AI search – the first result you see – is basically rubbish, often completely misleading. Sadly google’s search is still way above the competition’s overall, AFTER you skip over the monetised results on the first page that is.
On the other hand we have this: https://bgr.com/science/doctors-left-a-sick-man-for-dead-but-then-ai-saved-his-life/
not having the slightest inclination to dip me toes into that morass, i do wonder however if, by the sounds of it, they’ve been designed to emulate the (very) average maggat rally attendee, being vox popped by a daily show correspondent?
I did test ChatGPT a couple of times when it first appeared Clive but I don’t actually use google’s AI, I actively avoid it (along with Microsoft’s Copilot). Google’s search engine regularly chucks up an AI Overview as the first result without any prompting. You can’t turn that off but apparently there are filters and browser extensions that hide these unsolicited (and sometimes patently wrong) “overviews”.
I don’t know what algorithms they use to select AI answers but you could be right. They might defer to the most read page, or the most read site which pays google, or the least likely answer, or the answer most American Trump supporters would expect/be capable of understanding. Anythiimng’s possible given google is in the Trump tent along with a couple of handsful of other billionaires who prefer profit to ethics.
When you understand how these large language models work, you know why they are useless for anything filler text. They are a statistical model. Given a, b and c it predicts the next most likely word based on the data it has been fed. Its a simulation of a reply and, like all simulations, its fidelity is limited.
This is why they produce something like a C-student high school social studies report.