Thanks, these are some great concrete examples of how LLMs approach responding to their prompts.
It figures that, when pushed a bit beyond the most common kind of example, ChatGPT would label that bit on the end "a stop codon" when it's not even a codon; the relevant bits of its training data probably have a strong bias toward translations that do end in actual stop codons so it reached for that. With the prompt given, its response really was the "most likely" within the confines of its training. I think that's something that is being repeatedly misunderstood in media coverage and discussions of generative LLMs. They don't make true statements when they're working correctly and false ones when they're not; a "wrong answer" -- hallucination, confabulation, whatever we call it -- is really just a post-hoc label that's separate from the model. The model is giving the "right answer" so long as it's the most likely text to continue from where the prompt leaves off. Truth and fact aren't part of the system. I wish they'd be clearer on that point.
The "reinforcement learning from human feedback" (RLHF) applied to the models also seems to give them a neurotic predisposition to be agreeable and inoffensive (at least, I assume that tone comes from heavy use of RLHF; they all seem to share this behavior) so we end up with systems that bend over backwards to avoid ever telling us that we're wrong or that acknowledge inconsistencies. In Bard's case it sure looks to me like it generates an even more wrong answer while flailing around trying to take the blame yet still produce some kind of explanation. It's hard not to cringe at what comes across as a blend of incompetence and groveling in Bard's output. As far as tone goes, I'd almost rather have Bing chat's unhinged belligerence from a few months back compared to that.
