Who’d’ve thunk large language models would be good at language and not coding.

(I know, programming has languages too but there is far less available data on those languages used “talking” to each other.)

I’m learning coding right now in college and my professors are making sure we know how to effectively work with LLMs when coding.

It should never be used for generating code for anything other than examples. One should always remember it will answer what is asked and nothing more, and must make considerations for this when adapting generated code for your uses. Its best to treat it like a frontend for parsing documentation and translating it into simpler language, or translating a function in one language to another.

God, I hate Spring with a passion! Just pure black magic.

Also it’s just wrong a lot when I ask things I don’t know.

“Use private_key parameter instead of pkey.”
Alright, cool, good tip.
“Unknown parameter private_key.”
*jim face*

10×s developers who could produce 0 code without it

Let me see; ten times nothin’, add nothin’, carry the nothin’…

I like the article, but that one almost seems like a strawman. Is anyone actually arguing that?

I’ve been using the AI to help me with some beginner level rust compilation checks recently.

I never once got an accurate solution, but half the time it gave me a decent enough keyword to google or broken pattern to fix myself. The other half of the time it kept giving me back my own code proudly telling me it fixed it.

Don’t worry though, AGI is right around the corner. Just one more trillion dollars bro. One trillion and we’ll provide untold value to the shareholders bro. Trust me bro.

The RegEx thing is so true in my experience. I started working on a Neovim plugin to make editing injected code easier, and instead of suggesting Treesitter integration it wanted to create its own parser using RegEx…

Classic

Have you tried telling it not to suggest a regex solution?

I’ve never seen it recommend a solution using regex. And I’ve had it provide a lot of useful code. Perhaps you need to look into prompt engineering training?

it’s great at interpreting stack traces/logs/crash reports

How dare you! Can’t you see there’s a circle jerk in progress?

I’ve had “success” with using them for small one-off projects where I don’t care too much about correctness, efficiency, or maintainability. I’ve tried using various AI tools (Copilot, Cursor agents, etc) for more serious projects where I do care about those things, and it was counter-productive (as studies have shown).

Hmm, I was curious if ChatGPT still gives inefficient code when asking it to write quicksort in Python, and it still does:

def quicksort(arr):
    if len(arr) <= 1:  # Base case
        return arr
    pivot = arr[len(arr) // 2]  # Choose middle element as pivot
    left = [x for x in arr if x < pivot]   # Elements less than pivot
    middle = [x for x in arr if x == pivot] # Elements equal to pivot
    right = [x for x in arr if x > pivot]  # Elements greater than pivot
    return quicksort(left) + middle + quicksort(right)

That’s not really quicksort. I believe that has a memory complexity of O(n log n) on the average case, and O(n^2) for the worst case. If AI does stuff like this on basic, well-known algorithms, it’s likely going to do inefficient or wrong stuff in other places. If it’s writing something someone is not familiar with, they may not catch the problems/errors. If it’s writing something someone is familiar with, it’s likely faster for them to write it themselves rather than carefully review the code it generates.

Your response seems very enterprise-focused. I think you might be missing the kind of software development that happens before it becomes enterprise. All of these metrics are very reasonable for new products, startups, consulting, and hobby hackers. If code were moving 10X now, we should reasonably see 10X new growth. These numbers show we’re not.

Arguably we should also see a 10X something in legacy and enterprise as well which is harder to measure. If we assume a 10X dev is producing 10X more code, we should expect 10X more bugs so we should also see a rise in QA positions. We’re not, so that’s a good indicator. We should also see a rise in product manager roles to handle teams that are suddenly producing 10X per member. We’re not, so that’s a good indicator. We should also see 10X new product deliveries from companies like Salesforce. We’re not, so that’s a good indicator.

You completely missed the sections on how long these tools have been available. Your point about the internet would be valid if this article was written in, say, 2021 when Copilot and Tabnine were new and hot. It would also have maybe been valid in early 2023 when people were first spinning up workflows off ChatGPT and making 10X promises. It’s now years later and we’re not seeing any growth in any of those numbers as illustrated by the article.

The web did not exist until the 90’s.

That’s unfortunate for you, but not every company operates like that. There’s new startups every day, and with all the VC money they’re getting, they’re in a prime position to be building these new products. The fact so many AI products are under-delivering is enough of an indicator for me to agree with the article’s logic

Wow! There’s one with a brain over here!

How dare you!

Pretty much everyone I’ve talked to about this says the same thing. LLMs are useful for one-off scripts or quickly generating boilerplate. It just turns out that those tasks don’t make up the majority of programming work unless you are in a bullshit job anyway.

We aren’t yet great at knowing when LLM will save time and when it will inflate time.

If AI coding is causing such a revolution, why are we not seeing a sharp increase in code production?

(Backed up with data)