Newshot

Ask HN: Who's really winning with AI-Coding?

Original text(via HN AI)
Ask HN: Who's really winning with AI-Coding?
4 points by unclebucknasty 4 hours ago │ hide │ past │ favorite │ 4 comments
I've noticed the same pattern across three different Claude-coded projects now. Initial velocity is strong, but starts to fade until returns are negative. It seems a combination of changes in model behavior and increasing code base size—emphasis on the latter. The larger the code base and documented history/progress (e.g. through updates to completed work and instructions, as in .md files), the more drift starts to set in. At a certain point, you get collapse. This is where it can't maintain consistency within a single unit of work. It can manifest, for example, as adding a similar element to a view, but styling it in a completely different manner than existing elements. It literally feels random. It can also manifest as incoherent or poorly implemented logic and behavior. I've followed different advice, restructured projects, entered plan mode, added deterministic checks, styles, rules, etc. But, it feels like whack-a-mole and it's unclear that it actually adheres to these rules in the first place. Then, there's the question of it moving the problem. In response to my complaints of inconsistent UI generation, it recommended a style check to run after its UI changes. Of course, that begs the question of why it doesn't just apply those rules in the initial generation. But, I am at a place now where it spins up AI agents to do a full UI/UX review—Playwright and all. Yet, it's still drifting and not catching it. This is consistent with its unit tests—on my latest project it has now generated a suite of hundreds of somewhat time consuming tests that it constantly runs pre-commit, and not a single test has ever failed. Meanwhile, there have been a number of bugs. When questioned, it said it had been testing the "wrong layer" and had essentially generated self-confirming tests. My question is who is really winning here? When querying Google, it points to so many claims of success and approaches that supposedly contribute to that success. But, when you break down the underlying source for those claims, it appears to be largely Reddit. Yet, pressing it further reveals that virtually every actual study shows either little to no productivity gains or actual regression—particularly where experienced engineers are concerned. And some of those studies debunk the self-reported claims of exactly the kind you see on Reddit, etc. So, who here has developed a quality production application of any complexity using AI coding? If anyone, which harness/model, and what techniques did you use? Also, can you point us to your repo?

It seems a combination of changes in model behavior and increasing code base size—emphasis on the latter. The larger the code base and documented history/progress (e.g. through updates to completed work and instructions, as in .md files), the more drift starts to set in.

At a certain point, you get collapse. This is where it can't maintain consistency within a single unit of work. It can manifest, for example, as adding a similar element to a view, but styling it in a completely different manner than existing elements. It literally feels random.

It can also manifest as incoherent or poorly implemented logic and behavior. I've followed different advice, restructured projects, entered plan mode, added deterministic checks, styles, rules, etc. But, it feels like whack-a-mole and it's unclear that it actually adheres to these rules in the first place.

Then, there's the question of it moving the problem. In response to my complaints of inconsistent UI generation, it recommended a style check to run after its UI changes. Of course, that begs the question of why it doesn't just apply those rules in the initial generation.

But, I am at a place now where it spins up AI agents to do a full UI/UX review—Playwright and all. Yet, it's still drifting and not catching it. This is consistent with its unit tests—on my latest project it has now generated a suite of hundreds of somewhat time consuming tests that it constantly runs pre-commit, and not a single test has ever failed. Meanwhile, there have been a number of bugs. When questioned, it said it had been testing the "wrong layer" and had essentially generated self-confirming tests.

My question is who is really winning here? When querying Google, it points to so many claims of success and approaches that supposedly contribute to that success. But, when you break down the underlying source for those claims, it appears to be largely Reddit.

Yet, pressing it further reveals that virtually every actual study shows either little to no productivity gains or actual regression—particularly where experienced engineers are concerned. And some of those studies debunk the self-reported claims of exactly the kind you see on Reddit, etc.

So, who here has developed a quality production application of any complexity using AI coding? If anyone, which harness/model, and what techniques did you use? Also, can you point us to your repo?

fedpost 4 hours ago │ next [–] I think you hit the nail on the head. In my experience you can maintain a large codebase if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit but beyond that you're just pulling a slot machine handle and hoping for good results. That works well with small programs and sometimes people get lucky with the repeated coin flips on larger ones but eventually it breaks down. replyfedpost 4 hours ago │ next [–] I think you hit the nail on the head. In my experience you can maintain a large codebase if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit but beyond that you're just pulling a slot machine handle and hoping for good results. That works well with small programs and sometimes people get lucky with the repeated coin flips on larger ones but eventually it breaks down. reply
fedpost 4 hours ago │ next [–] I think you hit the nail on the head. In my experience you can maintain a large codebase if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit but beyond that you're just pulling a slot machine handle and hoping for good results. That works well with small programs and sometimes people get lucky with the repeated coin flips on larger ones but eventually it breaks down. reply
unclebucknasty 4 hours ago │ parent │ next [–] > in my experience...if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit Have you found the effort to review the reams of code it so easily produces significantly more productive than just writing the code yourself? And, given the frequently difficult to read/maintain code it produces, is it worth it—even if you are seeing some gains with code review time included? Just really hard to glean what the net impact really is, in order to calibrate expectations accordingly. replyunclebucknasty 4 hours ago │ parent │ next [–] > in my experience...if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit Have you found the effort to review the reams of code it so easily produces significantly more productive than just writing the code yourself? And, given the frequently difficult to read/maintain code it produces, is it worth it—even if you are seeing some gains with code review time included? Just really hard to glean what the net impact really is, in order to calibrate expectations accordingly. reply
unclebucknasty 4 hours ago │ parent │ next [–] > in my experience...if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit Have you found the effort to review the reams of code it so easily produces significantly more productive than just writing the code yourself? And, given the frequently difficult to read/maintain code it produces, is it worth it—even if you are seeing some gains with code review time included? Just really hard to glean what the net impact really is, in order to calibrate expectations accordingly. reply
fedpost 3 hours ago │ root │ parent │ next [–] It's...complicated? If you break things up well you can focus on your core code and let the LLMs handle the periphery (namely build system, dependencies, unit tests etc). I think the advantage lies in allowing the developer to focus on the parts that are critical and not the drudgery but that's not what most people are doing. replyfedpost 3 hours ago │ root │ parent │ next [–] It's...complicated? If you break things up well you can focus on your core code and let the LLMs handle the periphery (namely build system, dependencies, unit tests etc). I think the advantage lies in allowing the developer to focus on the parts that are critical and not the drudgery but that's not what most people are doing. reply
fedpost 3 hours ago │ root │ parent │ next [–] It's...complicated? If you break things up well you can focus on your core code and let the LLMs handle the periphery (namely build system, dependencies, unit tests etc). I think the advantage lies in allowing the developer to focus on the parts that are critical and not the drudgery but that's not what most people are doing. reply
proc0 3 hours ago │ prev [–] I've only been using some of the smaller but still capable local AI. I'm still skeptic on investing too much effort in the skills (my skills) needed for an AI-first workflow. I think one of the main pillars of engineering is being ignored, which is precision and predictability. Now I'm sure there is an engineering solution to this problem itself, meaning you can take the chaotic nature of AI workflows and build things around it to manage it, however this is not a trivial amount of overhead. I'm not convinced it is worth it, and in a way it defeats the purpose of AI to begin with, which is intelligent systems that would at least know when they are wrong and what their own limitations are. That said, AI already has a place in development in general just not as a main driver. If progress stays at this level, it's a matter of making it smaller, cheaper and local first, and then it will be an integral part of build tools and the like. So far I think the best use case is super-charging human driven development. replyproc0 3 hours ago │ prev [–] I've only been using some of the smaller but still capable local AI. I'm still skeptic on investing too much effort in the skills (my skills) needed for an AI-first workflow. I think one of the main pillars of engineering is being ignored, which is precision and predictability. Now I'm sure there is an engineering solution to this problem itself, meaning you can take the chaotic nature of AI workflows and build things around it to manage it, however this is not a trivial amount of overhead. I'm not convinced it is worth it, and in a way it defeats the purpose of AI to begin with, which is intelligent systems that would at least know when they are wrong and what their own limitations are. That said, AI already has a place in development in general just not as a main driver. If progress stays at this level, it's a matter of making it smaller, cheaper and local first, and then it will be an integral part of build tools and the like. So far I think the best use case is super-charging human driven development. reply
proc0 3 hours ago │ prev [–] I've only been using some of the smaller but still capable local AI. I'm still skeptic on investing too much effort in the skills (my skills) needed for an AI-first workflow. I think one of the main pillars of engineering is being ignored, which is precision and predictability. Now I'm sure there is an engineering solution to this problem itself, meaning you can take the chaotic nature of AI workflows and build things around it to manage it, however this is not a trivial amount of overhead. I'm not convinced it is worth it, and in a way it defeats the purpose of AI to begin with, which is intelligent systems that would at least know when they are wrong and what their own limitations are. That said, AI already has a place in development in general just not as a main driver. If progress stays at this level, it's a matter of making it smaller, cheaper and local first, and then it will be an integral part of build tools and the like. So far I think the best use case is super-charging human driven development. reply

In my experience you can maintain a large codebase if you are exceedingly disciplined and are actually reviewing the lines that the LLM is trying to edit but beyond that you're just pulling a slot machine handle and hoping for good results. That works well with small programs and sometimes people get lucky with the repeated coin flips on larger ones but eventually it breaks down.

Have you found the effort to review the reams of code it so easily produces significantly more productive than just writing the code yourself?

And, given the frequently difficult to read/maintain code it produces, is it worth it—even if you are seeing some gains with code review time included?

Just really hard to glean what the net impact really is, in order to calibrate expectations accordingly.

If you break things up well you can focus on your core code and let the LLMs handle the periphery (namely build system, dependencies, unit tests etc). I think the advantage lies in allowing the developer to focus on the parts that are critical and not the drudgery but that's not what most people are doing.

I think one of the main pillars of engineering is being ignored, which is precision and predictability. Now I'm sure there is an engineering solution to this problem itself, meaning you can take the chaotic nature of AI workflows and build things around it to manage it, however this is not a trivial amount of overhead. I'm not convinced it is worth it, and in a way it defeats the purpose of AI to begin with, which is intelligent systems that would at least know when they are wrong and what their own limitations are.

That said, AI already has a place in development in general just not as a main driver. If progress stays at this level, it's a matter of making it smaller, cheaper and local first, and then it will be an integral part of build tools and the like. So far I think the best use case is super-charging human driven development.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗