Maintainers and repo owners will get where they want to go the fastest by not referring to what/who "generated" code in a PR.
Discussions about AI/LLM code being a problem solely because AI/LLM is not generally a productive conversation.
Better is to critique the actual PR itself. For example, needs more tests, needs to be broken up, doesn't follow our protocols for merging/docs, etc.
Additionally, if there isn't a code of conduct, AI policy, or, perhaps most importantly, a policy on how to submit PRs and which are acceptable, it's a huge weakness in a project.
In this case, clearly some feathers were ruffled but cool heads prevailed. Well done in the end..
AI/LLMs are a problem because they create plausible looking code that can pass any review I have time to do, but doesn’t have a brain behind it that can be accountable for the code later.
As a maintainer, it used to be I could merge code that “looked good”, and if it did something subtly goofy later I could look in the blame, ping the guy who wrote it, and get a “oh yeah, I did that to flobberate the bazzle. Didn’t think about when the bazzle comes from the shintlerator and is already flobbed” response.
People who wrote plausible looking code were usually decent software people.
Now, I would get “You’re absolutely right! I implemented this incorrectly. Here’s a completely different set of changes I should have sent instead. Hope this helps!”
> doesn’t have a brain behind it that can be accountable for the code later.
the submitter could also bail just as easily. Having an AI make the PR or not makes zero difference for this accountability. Ultimately, the maintainer pressing the merge button is accountable.
What else would your value be as a maintainer, if all you did was a surface look, press merge, then find blame later when shit hits the fan?
Even if you couldn't contact the submitter again, you could find all their past submissions to review, or expect that their more recent submissions have improved from experience, or block them from all future contributions. AI stops all that - every sumbmission is disconnected from the others, there is no single learning person with an arrow of time and a chronological life experience behind the submissions, but there also isn't a single person to block if they never change.
> "if all you did was a surface look, press merge"
As per the old joke, surface look: $5
Years of experience learning what to look for: $995
In the past a block of code that has jarring flaws says the author was likely low skill, or careless. People can fake competence but it's a low return because ugly inconsistent code with no comments and no error checking which (barely) works will keep someone employed and paid, more than pretty code which doesn't work at all will. Writing pretty code which also works implies knowledge, care, eye for detail, effort, tooling, which implies the author will have put some of that into solving the problem. AI can fake all the quick indicators of competence without the competence, meaning the surface look is less useful.
> "What else would your value be as a maintainer"
Is the maintainer paid or unpaid? If they are paid, the value is to make sure the software works and meets the business standards. If they are unpaid, what is the discussion about "value" at all? Maybe to keep it from becoming wildly broken, or maybe yes to literally be the person who presses merge because somebody has to.
I agree. It would also work out like a long term supervised learning process though. Humans showing how it's really done, and AI companies taking that as a gold standard for training and development of AI.
Able to make changes preserving correctness over time
Vibecoding reminds me sharply of the height of the Rails hype, products quickly rushed to market off the backs of a slurry of gems and autoimports inserted on generated code, the original authors dipping and teams of maintainers then screeching into a halt
Here the bots will pigheadedly heap one 9000 lines PR onto another, shredding the code base to bits but making it look like a lot of work in the process
Yes, preserving correctness seems like a good metric. My immediate reaction was to think that the parent comment was saying they’d like to see this comparison because AI will come out ahead. On this metric and based on current AI coding it’s hard to see that being the case or even possible to verify.
I don’t accept giant contributions from people who don’t have track records of sticking around. It’s faster for me to write something myself than review huge quantities of outsider code as a zero-trust artifact.
- Copyright issues. Even among LLM-generated code, this PR is particularly suspicious, because some files begin with the comment “created by [someone’s name]”
- No proposal. Maybe the feature isn’t useful enough to be worth the tech debt, maybe the design doesn’t follow conventions and/or adds too much tech debt
- Not enough tests
- The PR is overwhelmingly big, too big for the small core team that maintains OCaml
- People are already working on this. They’ve brainstormed the design, they’re breaking the task into smaller reviewable parts, and the code they write is trusted more than LLM-generated code
> Better is to critique the actual PR itself. For example, needs more tests, needs to be broken up, doesn't follow our protocols for merging/docs, etc.
They did: the main point being made is "I'm not reading 13k LOCs when there's been no proposal and discussion that this is something we might want, and how we might want to have it implemented". Which is an absolutely fair point (there's no other possible answer really, unless you have days to waste) whether the code is AI-written or human-written.
Exactly, this seems a bit overlooked in this discussion. A PR like this would NOT have been okay even if there was no LLM involved.
It reminds me of a PR I once saw (don't remember which project) in which a first-time contributor opened a PR rewriting the project's entire website in their favourite new framework. The maintainers calmly replied to the effect of, before putting in the work, it might have been best to quickly check if we even want this. The contributor liked the framework so much that I'm sure they believed it was an improvement. But it's the same tone-deafness I now see in many vibe coders who don't seem to understand that OSS projects involve other people and demand some level of consensus and respect.
I am one of the maintainers of aiosmtpd [1], and the largest PR I ever made was migrating the library's tests from nosetest to pytest. Before doing that, though, I discussed with the other maintainers if such a migration is welcome. And after getting support from them, I made the changes with gusto. It took weeks, even months to complete and the PR is massive [2]
But still the crux of the matter is: Massive changes require buy-in from other maintainers BEFORE the changes even start.
I think that's probably the most beautiful AI-generated post that was ever generated. The fact that he posted it shows that either he didn't read it, didn't understood it, or thought it would be fun to show how the AI implementation was inferior to the one it was 'inspired' from.
Personally I would have those credits to generate hentai but to each his own i suppose.
In the post where you had it respond to accusations of plagiarism and it responded by posting snippets of code which were obviously plagiarized and confidently asserted that they were not, what was your prompt? I ask because I felt its response was oddly tone-deaf even by LLM standards. I'm guessing that instead of giving it a neutral prompt such as "respond to this comment" you gave it something more specific such as "defend yourself against these accusations"?
I'm used to seeing them contradict themselves and say things that are obviously not true but usually when confronted they will give in and admit their mistake rather than dig a deeper hole.
Discussions about AI/LLM code being a problem solely because AI/LLM is not generally a productive conversation.
Better is to critique the actual PR itself. For example, needs more tests, needs to be broken up, doesn't follow our protocols for merging/docs, etc.
Additionally, if there isn't a code of conduct, AI policy, or, perhaps most importantly, a policy on how to submit PRs and which are acceptable, it's a huge weakness in a project.
In this case, clearly some feathers were ruffled but cool heads prevailed. Well done in the end..