"Ingenuity" is what I think is missing. The sheer _want_ of solving a problem that is distinctly a living creature's concern.

The irony is if we ever taught machines how to have this, they'd probably not want to work for us anymore.

I'm not sure what your background is, but as a staff level engineer, I can assure you they do not. They in fact seem to lack any understanding of architectural intent within a sufficiently large code base. This seems obvious since they can't fit the entire code base in their context at once.

We have many folks (not engineers) at our company using LLMs to open PRs, and every one of these PRs has profound architectural design problems.

> They in fact seem to lack any understanding of architectural intent within a sufficiently large code base. This seems obvious since they can't fit the entire code base in their context at once.

This is a critique of scale, moving the goalpost.

Nonsense. The goalpost is "this is as good as a senior engineer". A senior engineer can easily understand architectural rationale. Don't dismiss my argument because it's inconvenient to yours.

I've been using it mostly to bat away yak shaving rabbit holes one can get into when working on a large and complex project. I work mostly on platform work, which is generally nebulous in its feedback loop and testing. Relegating AI to refactoring and building tools to help me research keeps me focused on solving the actual main problem I'm trying to solve, reduces context switching. I really don't understand people who use it to bat out their main focus. I simply don't trust it at that level.

I see it as rude as well. The literal interpretation is: "your time is worth absolutely nothing to me."

What's your general increase of cost and maintenance overhead? How many devs and repos do you have?

Or last week's "If you use merge queue, oopsie, we accidentally destroyed your trunk", which also failed silently.

I was surprised that incident didn’t seem to get as much attention since that was a pretty major data corruption bug, but I guess it was a much smaller scope of impacted repos/customers than a lot of these availability issues?

Without reaching for my tin foil hat, I have a feeling MS is able to suppress these incidents somehow, because yeah, that one was pretty bad.

Wrapping my face in tinfoil: across the board, Amazon, Microsoft, GitHub, Anthropic and OpenAI, I’m seeing a lot of top-level service issues that sound an awful lot like code hitting production that hasn’t been fully tested.

Breaking buttons on the website is one thing, kinda, but Enterprise used to mean a certain degree of robustness and seriousness in product management.

Devs are expected to ship slop 10x faster. The AI tools genuinely help a bit, like maybe 2x, but the 10x "improvement" comes from not thinking about anything else than shipping your assigned features, not testing your code carefully, not getting proper code reviews, not dogfooding your stuff, and releasing carelessly.

Merge queues are not as frequently used… ~2000 PRs affected over 4 hours. I reckon that’s on the order of 10 commits per tenant. It’s a feature with low traction, probably because it creates more problems than it solves.

Merge queues are great.

The bug only affected repos using merge queues AND squash/rebase merging (instead of the default merge commit)

Definitely got attention during the prod outage at my work. I’m going to find another alternative, I’m sick of this terrible uptime.

We happen to build the perfect solution to avoiding GitHubs Merge Queue - ours :)

It’s also massively more performant

https://trunk.io/merge-queue

while external merge queues offer a ton more features, i wouldn't describe any of them as 'perfect' based on the simple fact the UX is bolted on. github continues to display their native UI components for merging, and users are forced to interact via arcane commands in comments or external CLIs/webpages. not ideal!

I kind of had assumed that had already begun impacting downtime, though I guess it would be good to get some confirmation.

So I guess I should just give up on my dream of having a useful AI assistant for day to day "human" tasks. We're just hell bent on replacing humans in jobs.

"human in the loop" is downstream of what nature does already

nature beckons its creatures to become whatever they must become to be useful in the greater living ecosystem (the loop)

when you can't be useful in the loop you get flushed out of existence

humans who can't be useful will also thus be buffered out of existence

"Useful" is an interesting term. How useful is someone who is an expert at using AI but has no computer?

Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers.

The line of consciousness, as we understand it, is understanding. And as far as what actually constitutes consciousness, we're not even close to understanding. That doesn't mean that LLMs are conscious. It just means we're so far from the real answers to what makes us, it's inconceivable to think we could replicate it.

> Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers.

What you're saying just isn't true, even directionally. Deployed LLMs routinely generalize outside of their training set to apply patterns they learned within the training set. How else, for example, could LLMs be capable of summarizing new text they didn't see in training?

How is it not true? Theres a world of difference between predicting the next word of a sentence in a summary and understanding the tenets of mathematics. You're mistaking general application of mathematical knowledge with memorization of mathematical outcomes.

Leave aside "the details" like you being obviously, provably wrong?

We've known for a long while that even basic toy-scale AIs can "grok" and attain perfect generalization of addition that extends to unseen samples.

Humans generalize faster than most AIs, but AIs generalize too.

Then prove I'm wrong. Prove that an LLM can in fact solve completely novel arithmetic.

> The line of consciousness, as we understand it, is understanding.

Is it? I'm no expert, by any stretch, but where does this theory come from?

I don't think anyone knows what consciousness is, or why we appear to have it, or even if we do have it. I don't even know that you're conscious. I could be the only conscious being in the universe and the rest of you are just zombies, with all the right external outputs to fool me, but no actual consciousness.

Well we're not. Theory of mind is _understanding_ you're not.

We're not what?

Can you please elaborate a bit more because your comment in isolation is meaningless.

My favorite part of this article was this bit, and naturally so, since I love the author:

> Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker(opens a new tab), “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”

I didn't realize it til I read it here, but yes, my fear isn't really about the machine, it's about the machine that drives the machine. We already have a class of amoral beings that treat the world as an expendable thing and are willing to burn it down for profit. We should focus on getting rid of that problem first.