LLMs are decent with coding tasks if you know what you’re doing

Only if the thing you are trying to do is commonly used and well documented, but in that case you could just read the documentation instead and learn a thing yourself, right?

The other day I tried to get some instructions on how to do something specific in a rather obscure and rather opaquely documented cli tool that I need for work. I couldn’t quite make sense of the documentation, and I found the program’s behavior a bit erratic, so that’s why I turned to AI. It cheerfully and confidently told me (I’m paraphrasing): oh to do “this specific thing” you have to use the --something-specific switch, and then it gave some command line examples using that switch that looked like they made complete sense.

So I thought: oh, did I overlook that switch? Could it be that easy? So I looked in the documentation and sure enough… the AI had been bullshitting me and that switch didn’t exist.

Then there was the time when I asked it to generate an ARM template (again, poorly documented bullshit) to create some service in Azure with some specific parameters. It gave me something that looked like an ARM template, but sure as hell wasn’t a valid one. This one wasn’t completely useless though, at least I was able to cross reference with an existing template and with some trial-and-error, I was able to copy over some of the elements that I needed.