Hate to break it to you, but many kids actually do better away from their parents than with them.
It's extremely sad, but a consistent finding in early childhood education is that the children who thrive most in daycares tend to come from the least advantaged backgrounds.
So a policy of paying parents to stay home would mostly benefit kids who are already well off.
Kids are social and like playing and learning from other kids. Daycare lets them do just that. It’s a great thing and every toddler I’ve met who wasn’t in daycare was behind in something. Especially verbal skills.
Plus daycare allows women to continue their career progression. It’s soo important. Not every woman wants to end their career as a mother to a young kid. Daycare enables successful women to thrive and still have families.
Your anecdote is just that. All of it is highly dependent on the child, their environment, and the 'educator'. Please don't make assumptions based on your limited exposure; it's not helpful.
Your "it depends" argument is that some kids aren't social, don't like playing with other kids, are better off not having exposure to social interaction with peers and practice talking.
If this is the criticism then it's a glowing endorsement of daycare and school.
No; it depends on the 'educator'. A daycare that doesn't have kids interacting in a positive way could be just as detrimental as a parent that doesn't socialize their children externally to the home.
I'm just gonna throw this out here: Well-off kids who barely know their workaholic parents have different but equally bad issues for society, than the poor kids do.
Those poor kids have learning deficits. The "well-off" kids often have morality deficits.
A mom or dad raising them properly might help them more than being Student #642 in a government childcare facility.
This isn't an argument against childcare. My children attended preschool for 3 years before Kindergarten. But I'd rather that people got equal support to have a stay-at-home parent so that people can choose.
From what I’ve seen, the research leans the other way. For example:
Children from more advantaged families were actually more likely to view unfair distribution as unfair, while poorer children were more likely to accept it. [0]
Mother’s work hours show no link to childhood behavioral problems, it’s schedule flexibility that matters. [1]
For working-class families, more father work hours correlated with fewer behavioral problems.[2]
The idea that “well-off kids” end up with morality deficits because their parents work a lot doesn’t seem to hold up.
You aren't wrong but calling it being "Student #642 in a government childcare facility" the wrong way of looking at it. Children grow up best when they are allowed to play with other children. Modern society robs kids of that and helicopter parents are bad for society.
I agree with you vigorously on both those points. I am skeptical however that NM will be able to create a lot of healthy, play-based environments for so many kids.
The market already has incentives to create them -- a ton of good places have waiting lists nationwide, showing unmet demand even at the current price. This suggests the price will need to go higher to attract enough people to do this job. It seems their "$12,000 value" estimate is based on an optimistic belief that they will be buying childcare for their citizens at current prices. When they realize there aren't that many slots available at current rates of pay, will they be okay significantly increasing the costs of the program?
So, my expectations for these facilities are very low and that's a big part of my concern.
> Hate to break it to you, but many kids actually do better away from their parents than with them.
Is this based on something?
There's research left and right shows that children under 36 months at group nurseries are linked to increased aggression, anxiety, lower emotional skills, elevated cortisol (stress hormone), which is associated with long-term health and developmental risks.
Infants and children do better with one-to-one care at home by their parents and familiar faces, rather than strangers in a group setting.
Perhaps there is something about the environment of an economically disadvantaged household that could be improved by a stipend which allows at least one parent the breathing room to dedicate full time attention to the child instead of a job (or multiple jobs). I don't think the findings you mentioned cut against that idea at all.
I hear you saying the benefit of dedicated caregiving for children mostly helps families with less economic advantage. I'd agree with that, and suggest that OP's proposal capitalizes on exactly that. I'm not convinced of what may be implied in your argument that low-earners make for bad parents and that children should be separated more from their parents for their own good. Let the internal dynamics of a family be solved first, before saying we need to separate parents from children more.
Moreover, those with more economic advantage are unlikely to take a stipend in exchange for staying home. That's not a good deal when keeping the job pays so much that they can afford to pay for childcare.
It is precisely those with less advantage who will take the deal.
So I don't agree with your prediction that such a stipend mostly benefits those who are already well off.
My daycare was called preschool. It allowed my mother to focus on my infant brother during the day while I was literally two blocks away running around, coloring and learning shapes. Show and tell was my favorite.
The most obvious example is the children of addicts. It’s hard to imagine a kid is better off stuck at home with druggie parents than spending the day in daycare.
A good example of bottom quintile policy. Because the bottom quintile has a better outcome with a certain approach, it becomes standard care for everyone else.
A realistic stay-at-home subsidy would max out around $30k. Your proposal only meaningfully shifts incentives for the bottom income quintile. For everyone else:
- Upper-income families can already afford to choose whatever setup they want.
- Middle-income families couldn’t take it because it’d mean too steep a drop in income.
So the alternative you proposed economically benefits the bottom quintile while leaving their kids worse off. For everyone else, it probably either doesn't matter or gives them cash they don't need as much.
Another day, another person not getting discounted cash flow.
Models trained in 2025 don’t ship until 2026/7. That means the $3bn in 2025 training costs show up as expense now, while the revenue comes later. Treating that as a straight loss is just confused.
OAI’s projected $5bn 2025 loss is mostly training spend. If you don’t separate that out with future revenues, you’re misreading the business.
And yes, inference gross margins are positive. No idea why the author pretends they aren’t.
As a researcher, I can totally agree, but at the same time this isn't super straight forward. Things get weird because you can't just translate from one GPU to another. There isn't a clean calculation for that. There's also other issues like parallelism. Sure, your model is stable with a batch size of 8192 but that's across 1 node, it might not be stable with that batch across 2 nodes. This is a real frustrating part and honestly I don't think most people even are aware such issues exist.
Right now I'm just happy when people are including parameter, GMACs (or FLOPs), and throughput. I always include those and the GPUs I used. I also frequently include more information in the appendix but frankly when I include it in the front matter the paper is more likely to be rejected.
I can tell you why this isn't happening though. There's a common belief that scale is all you need. Which turns into "fuck the GPU poor". I've published works where my model is 100x smaller (with higher throughput, and far lower training costs), and the responses from reviewers tend to be along the lines "why isn't it better?" or "why not just distill or prune a large model?" There's this weird behavior that makes the black box stay a black box. I mean Yi Tay famously said "Fuck theorists" on twitter
Calculators are a very narrow form of intelligence as compared to the general-purpose intelligence that LLMs are. The muscle/steroid analogy from this same discussion thread is apt here. Calculators enhanced and replaced just one 'muscle', so the argument against them would be like "ya but do we really need this one muscle anymore?", whereas with LLMs the argument is "do we really even need a body at all anymore?" (if extrapolated out several more years into the future).
The main limitation of tokenization is actually logical operations, including arithmetic. IIRC most of the poor performance of LLMs for math problems can be attributed to some very strange things that happen when you do math with tokens.
I'd like to see a math/logic bench appear for tokenization schemes that captures this. BPB/perplexity is fine, but its not everything.
You right to left tokenize in groups of 3, so 1234567 becomes 1 234 567 rather than the default 123 456 7. And if you ensure all 1-3 digits groups are in the vocab, it does much better.
I suppose it is. There is a lot to tokenization - pre-tokenization, how to handle digits, the tokenization training approach - that is about adding cleverness. In the long run, the bitter lesson would be to just get rid of it all and learn from more data. Many people would love to do it. But I think for the case of BLT, digits will still be an issue. There is no way an autoregressive entropy model will be able to split numbers sensibly, since it has no idea how many digits are coming. It seems like it will struggle more with arithmetic. Perhaps you could reverse all the digits in a number, then it has a chance. So 12334 becomes 43321, and it gets to start from the ones digit. This has been suggested as an approach for LLM's.
Math operations go right to left in the text, while we write them left to right. So if you see the digits 123... in an autoreressive manner, you don't know really anything, since it could be 12345 or 1234567. If you flipped 12345 as 543..., you know the place value of each. You know that the 5 you encounter first is in the ones place, the 4 is the tens place, etc. It gives the LLM a better chance of learning arithmetic.
so basically reverse notation has the advantage of keeping magnitude of numbers (digits!) relative to each other constant (or at least anchored to the beginning of the number)
doesn't attention help with this? (or, it does help, but not much? or it falls out of autoregressive methods?)
Attention does help, which is why it can learn arithmetic, even with arbitrary tokenization. However, if you put it in a standard form, such as right-to-left groups of 3, you make it an easier problem for the LLM to learn. All the examples it sees are in the same format. Here, the issue is that BLT operates in an autoregressive manner (strictly left to right), which makes it harder to tokenize the digits in a way that is easier for the LLM to learn. Each digit is its own token (Llama style), or flipping the digits might be the best.
> Isn't that the opposite of the bitter lesson - adding more cleverness to the architecture?
The bitter lesson is that general methods and a system that learns trumps trying to manually embed/program human knowledge into the system, so clever architecture is ok and expected.
Ok great! This is precisely how I chunk numbers for comparison. And not to diminish a solid result or the usefulness of it or the baseline tech: its clear that it we keep having to create situation - specific inputs or processes, we're not at AGI with this baseline tech
DAG architectures fundamentally cannot be AGI and you cannot even use them as a building block for a hypothetical AGI if they're immutable at runtime.
Any time I hear the goal being "AGI" in the context of these LLMs, I feel like listening to a bunch of 18th-century aristocrats trying to get to the moon by growing trees.
Try to create useful approximations using what you have or look for new approaches, but don't waste time on the impossible. There's no iterative improvements here that will get you to AGI.
It doesn't feel particularly interesting to keep dismissing "these LLMs" as incapable of reaching AGI.
It feels more interesting to note that this time, it is different. I've been watching the field since the 90s when I first dabbled in crude neural nets. I am informed there was hype before, but in my time I've never seen progress like we've made in the last five years. If you showed it to people from the 90s, it would be mind blowing. And it keeps improving incrementally, and I do not think that is going to stop. The state of AI today is the worst it will ever be (trivially obvious but still capable of shocking me).
What I'm trying to say is that the shocking success of LLMs has become a powerful engine of progress, creating a positive feedback loop that is dramatically increasing investment, attracting top talent, and sharpening the focus of research into the next frontiers of artificial intelligence.
>If you showed it to people from the 90s, it would be mind blowing
90's? It's mind blowing to me now.
My daily driver laptop is (internally) a Thinkpad T480, a very middle of the road business class laptop from 2018.
It now talks to me. Usually knowledgeably, in a variety of common languages, using software I can download and run for free. It understands human relationships and motivations. It can offer reasonably advice and write simple programs from a description. It notices my tone and tries to adapt its manner.
All of this was inconceivable when I bought the laptop - I would have called it very unrealistic sci-fi. I am trying not to forget that.
Personally I'm hoping for advancements that will eventually allow us to build vehicles capable of reaching the moon, but do keep me posted on those tree growing endeavors.
It's about the immutability of the network at runtime. But I really don't think this is a big deal. General-purpose computers are immutable after they are manufactured, but can exhibit a variety of useful behaviors when supplied with different data. Human intelligence also doesn't rely on designing and manufacturing revised layouts for the nervous system (within a single human's lifetime, for use by that single human) to adapt to different settings. Is the level of mutability used by humans substantially more expressive than the limits of in-context learning? what about the limits of more unusual in-context learning techniques that are register-like, or that perform steps of gradient descent during inference? I don't know of a good argument that all of these techniques used in ML are fundamentally not expressive enough.
LLMs, considered as a function of input and output, are not immutable at runtime. They create tokens that change the function when it is called again. That breaks most theoretical arguments
But that view is wrong, the model outputs multiple tokens.
The right alternative view is that it's an immutable function from prefixes to a distribution over all possible sequences of tokens less than (context_len - prefix_len).
There are no mutable functions that cannot be viewed as immutable in a similar way. Human brains are an immutable function from input sense-data to the combination (brain adaptation, output actions). Here "brain adaptation" doing a lot of work, but so would be "1e18 output tokens". There is much more information contained within the latter
The claim was that it isn't possible in principle for "DAGs" or "immutable architectures" to be intelligent. That statement is confusing some theoretical results that aren't applicable to how LLMs work (output context is mutation).
I'm not claiming that compute makes the m intelligent. I'm pointing out that it is certainly possible, and at that level of compute it should be plausible. Feel free to share any theoretical results you think demonstrate the impossibility of "DAG" intelligence and are applicable
I am not saying it is impossible, I am saying it might be possible, but far from plausible with the current approach of LLMs in my experience with them.
That doesn't matter, are you familiar with any theoretical results in which the computation is somehow limited in ways that practically matter when the context length is very long? I am not
What do the vector space embeddings for digit strings even look like? Can you do arithmetic on them? If that's even desirable that it seems like you could just skip "embedding" altogether and intern all the numbers along one dimension.
Even if LLMs get better at arithmetic, they don't seem like the right tool for the job.
LLMs might never be able to crunch numbers reliably, however I expect they should be very good at identifying the right formula and the inputs for a problem ("i need the answer to x*y, where x=12938762.3 and y=902832.2332"). Then they can call a math engine (calculator or wolfram alpha or whatever) to do the actual computation. That's what humans do anyway!
Models are deterministic, they're a mathematical function from sequences of tokens to probability distributions over the next token.
Then a system samples from that distribution, typically with randomness, and there are some optimizations in running them that introduce randomness, but it's important to understand that the models themselves are not random.
The LLMs are deterministic but they only return a probability distribution over following tokens. The tokens the user sees in the response are selected by some typically stochastic sampling procedure.
Assuming decent data, it won't be stochastic sampling for many math operations/input combinations. When people suggest LLMs with tokenization could learn math, they aren't suggesting a small undertrained model trained on crappy data.
I mean, this depends on your sampler. With temp=1 and sampling from the raw output distribution, setting aside numerics issues, these models output nonzero probability of every token at each position
A large model well trained on good data will have logits so negative for something like "1+1=" -> 3 that they won't come up in practice unless you sample in a way to deliberately misuse the model.
This is only ideally true. From the perspective of the user of a large closed LLM, this isn't quite right because of non-associativity, experiments, unversioned changes, etc.
It's best to assume that the relationship between input and output of an LLM is not deterministic, similar to something like using a Google search API.
yep, even with greedy sampling and fixed system state, numerical instability is sufficient to make output sequences diverge when processing the same exact input
We passed 'mediocre' a long time ago, but yes, it would be surprising if the same vocabulary representation is optimal for both verbal language and mathematical reasoning and computing.
To the extent we've already found that to be the case, it's perhaps the weirdest part of this whole "paradigm shift."
thanks to training data + this being a popular benchmark, they're pretty good at grinding through symbolic mathematical derivations, which is often useful if you want an explanation of a mathematical concept. there's not really a better tool for this job, except for "a textbook which answers the exact question you have".
but from time to time, doing this does require doing arithmetic correctly (to correctly add two exponents or whatever). so it would be nice to be able to trust that.
i imagine there are other uses for basic arithmetic too, QA applications over data that quotes statistics and such.
> but from time to time, doing this does require doing arithmetic correctly (to correctly add two exponents or whatever). so it would be nice to be able to trust that.
It sounds weird, but try writing your problem in LaTeX - I don’t know why, I’ve found a couple models to be incredibly capable at solving mathematical problems if you write them in LaTeX.
regarding “math with tokens”: There was paper with tokenization that has specific tokens for int numbers, where token value = number. model learned to work with numbers as numbers and with tokens for everything else... it was good at math. can’t find a link, was on hugginface papers
Shouldn't production models already do this? They already tend to use tokenizers with complex rules to deal with a lot of input that would otherwise be tokenized in a suboptimal way. I recall a bug in an inference engine (maybe llama.cpp?) because of an implementation difference in their regex engine compared to the model trainer. Which means that the tokenizer used regex-based rules to chop up the input.
in paper mentioned “number” is a single sort-of “token” with numeric value, so network dealing with numbers like real numbers, separately from char representation. All the math happens directly on “number value”. In majority of current models numbers are handled like sequences of chars
It's not strange at all. I am playing with lambda calculus and combinatory logic now, as a base for mathematics (my interest is to understand rigorous thinking). You can express any computation using just S and K combinators, however, there is a price to that - the computations will be rather slow. So to make the computation faster, we can use additional combinators and rules to speed things up (good example is clapp() function in https://github.com/tromp/AIT/blob/master/uni.c).
Of course, the extra rules have to be logically consistent with the base S and K combinators, otherwise you will get wrong result. But if the inconsistent rule is complicated enough to be used only infrequently, you will still get correct result most of the time.
Which brings me to LLMs and transformers. I posit that transformers are essentially learned systems of rules that are applied to somewhat fuzzily known set of combinators (programs), each represented by a token (the term being represented by the embedding vector). However, the rules learned are not necessarily consistent (as it happens in the source data), so you get an occasional logical error (I don't want to call it hallucination because it's a different phenomenon from nondeterminism and extrapolation of LLMs).
This explains the collapse from the famous paper: https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin... One infrequent but inconsistent rule is enough to poison the well due to logical principle of explosion. It also clearly cannot be completely fixed with more training data.
(There is also analogy to Terry Tao's stages of mathematical thinking: https://terrytao.wordpress.com/career-advice/theres-more-to-... Pre-rigorous corresponds to soomewhat random set of likely inconsistent logical rules, rigorous to small set of obviously consistent rules, like only S and K, and post-rigorous to a large set of rules that have been vetted for consistency.)
What is the "solution" to this? Well, I think during training you somehow need to make sure that the transformer rules learned by the LLM are logically consistent for the strictly logical fragment of the human language that is relevant to logical and programming problems. Which is admittedly not an easy task (I doubt it's even possible within NN framework).
When humans get stuck solving problems they often go out to acquire new information so they can better address the barrier they encountered. This is hard to replicate in a training environment, I bet its hard to let an agent search google without contaminating your training sample.
