Why do I have 244gb of environments and beat myself up when I delete something to play a game?
This deep desire to affect change in a controllable way...
This infinite desire for self value defined by external validation.
It's not sustainable.
Perfection can only be obtained by observation of perfection of combined self through self and other.
It's okay to discard parts of yourself to balance yourself with your counterpart. A willing violation is no longer a violation.
Not observation of one or the other on a pedestal, but accepting that both are vital parts to the system and observing the perfection that comes from co-iteration.
Copyright infringement I guess. Other ideas could be passed off as a combination of several sources. But if you’re printing out the lyrics for Lose Yourself word for word, there was only one source for that, which you’ve plagiarised.
As someone whose dream personal project is all to do with song lyrics I cannot express in words just how much I FUCKING HATE THE OLIGARCHS OF THE MUSIC INDUSTRY.
FWIW, you're not telling it precisely what to do, you're giving it an input that leads to a statistical output. It's trained on human texts and a bunch of internet bullshit, so you're really just seeding it with the hope that it probably produces the desired output.
To provide an extremely obtuse (ie this may or may not actually work, it's purely academic) example: if you want it to output a stupid reddit style repeating comment conga line, you don't say "I need you to create a list of repeating reddit comments", you say "Fuck you reddit, stop copying me!"
Sure, but it's still a statistical model, it doesn't know what the instructions mean, it just does what those instructions statistically link to in the training data. It's not doing perfect forward logic and never will in this paradigm.
The fine tuning process isn't itself a statistical model, so that principle doesn't work on it. You beat the model into shape until it does what you want (DPO and varieties of that) and you can test that it's doing that.
Recipes can't be copyrighted but the text describing a recipe can. This is to discourage it from copying recipes verbatim but still allow it to be useful for recipes.
I don't have a large amount of time to devote to this.
But no, google is not out.
They just choke slammed ChatGPT with the Gemini announcement.
10m context with that retrieval rate is such a monstrous leap.
And to top it off, we got LargeWorldModel in the same week, capable of 1M token context with insane retrieval rate in the open source space.
So not only is the open source world currently technically ahead of ChatGPT, so is Google.
Which is why they had to announce SORA, because google's model is so far ahead of the competition.
That's also why it will probably be ages before we get access to SORA.
Now don't get me wrong, the average person can't afford 32 TPU's to run LWM, but we already have quants for it, which is a step towards enabling the average person (that somehow has 24-48gb of VRAM to get a taste of that power).
What is also striking is the fact that the new models are all multimodal as a standard. We not only leapfrogged in context size, but also in modalities.
The model seems to only benefit from having more modalities to work with.
I think the statement Bill Gates made claiming that "LLM's have reached a plateau" itself indicates they don't believe they can make more money from training better/larger models. Which indicates that they already did as well as they could with their existing people, and are now "years" behind google.
I never thought google could catch up, especially after their infamous "We have no moat" situation. But it seems they actually doubled down and did something about it.
To a lot of people, last Thursday was a very nihilistic day for Local Models, as the goalposts shifted from 128-200k context to 10M tokens with near perfect retrieval. It's literally insanely scary.
But luckily we got LWM, and that means we have only been 10xed.
Now the local people will work on figuring out how to bridge the gap, before being leapfrogged again. What is really insane is that, we have had LLAMA2 for over a year now, and nobody else figured out how to get this result from it, despite it being around so long.
I still believe there are modifications to the architecture of MoE that will unlock new powers that we haven't even dreamed of yet.
Sorry, this was supposed to be well thought out, but it turned more into stream of consciousness, and I honestly had no intention of disagreeing with you.
> But luckily we got LWM, and that means we have only been 10xed.
If I remember the paper correctly, it was something about a 4M context in there. So not 10x, but 2.5x.
> What is really insane is that, we have had LLAMA2 for over a year now, and nobody else figured out how to get this result from it, despite it being around so long.
This isn't true. For now, the task of extending context to 10M tokens is brute-forced by money (increased HW requirements for training and inference and increased training time are also a financial domain). And for now, there simply is no leapfrogging solution for open source or commercial models, which will decrease the costs by orders of magnitude.
We developed a tool that lets you kind of understand other people if the audio quality is sufficient.
With further training we can openly communicate with each other without fear of misunderstanding.
>The question is it reasonable for English speakers with a generally European heritage to be the stewards of niche world cultures?
Something better than nothing? It's very unlikely that a tribe in Africa or Polynesia is going to create their own ML model anytime soon.
Although if you read ML papers, it's usually pretty diverse. The article is hinting at colonialism, especially as experienced by the Maori, i.e. largely "English speakers with West European/British heritage, also male". However, the original Whisper paper has 6 authors out of which 2 are East Asian, 2 are of Jewish heritage, 1 is a native Russian speaker, 1 woman. I'm not sure how they're related to colonialism in NZ.
The article reads like the author of the article has what's called "gatekeeper syndrome" in my country, i.e. they're just upset someone trains ML models on Maori language without their official approval/oversight because the author feels their foundation is entitled to speak for all the Maori.
I wouldn't consider the ai models to be the stewards of the English language. The people who speak English are. When you hear a bot speak in broken English, you recognize that it's broken English.
The stewards on the language will similarly hear the bot as broken, and people who don't know the language won't know the difference.
2030: Deepfakes are rampant, causing significant issues in politics, entertainment, and personal privacy. However, instead of technological solutions, there is a growing trend of regulatory capture. Large corporations and governments begin to argue that deepfakes are an inevitable part of the digital landscape. The cost-effectiveness of creating deepfake content compared to traditional media production becomes a significant talking point.
2031: As deepfake technology becomes more sophisticated and cheaper, it starts to replace traditional media production methods. Major studios and media companies lobby for and receive regulatory approval to use deepfakes as a legitimate form of content creation. This shift is justified by the reduced cost and logistical ease of using AI-generated characters instead of real actors.
2032: A scandal arises involving a particularly damaging deepfake, but instead of driving a push towards authenticity verification technologies, it leads to further normalization of deepfakes. The argument is made that since distinguishing between real and fake content is increasingly difficult, society should adapt to accepting deepfake content as a new norm.
2033: Social media platforms and news outlets begin to openly embrace deepfake technology, citing cost reduction and the ability to generate more engaging content. Traditional media actors and creators are increasingly marginalized, with deepfake creators dominating the market.
2034: Regulatory bodies, heavily influenced by big tech and media conglomerates, begin to actively promote deepfake content. New regulations make it easier for deepfake content to be produced and disseminated, while traditional media production is bogged down by increased costs and regulatory hurdles.
2035: The public gradually accepts deepfakes as the primary form of digital content. Traditional media, with real actors and genuine locations, becomes a niche market due to its higher production costs and complexity. Deepfakes evolve in unexpected ways, permeating every aspect of digital media and blurring the line between reality and AI-generated content.
Anecdotally, I would like to mention that my partner has been sick for over a month with some kind of lung shit he got from an e-cigarette (vuse something disposable crap).
He has been literally unable to sleep through a night for over a month, multiple doctor visits, 3 courses of antibiotics, lung xray that apparently confirmed it was viral pneumonia, but yeah. I have been smoking for over 6 years and still alive, but he smoked that shit for less than a few weeks and he is barely functional.
I can't remember the last time I got a full night's sleep anymore, because I constantly have to apply muscle relaxants and ensure he has proper medication to be able to remain asleep.
So yeah, not really the point of your argument, but fuck e-cigarettes, for the rest of time I will have this stance. And yes, I am brutally aware of what cigarettes do as well, but don't knock the "theoretical" effects of e-cigarettes.
As said, another example of cherry-picking. They smoked a vape, therefore it must be vapes. They got the vaccine, therefore it must be vaccines. Rinse and repeat for whichever argument.
What is worse, pointing out these selective facts generally makes people become defensive, so they dive into searching for more cherry-picked arguments as ammunition to prove themselves right, rather than accepting that they might actually be wrong and objectively looking for information. It is the core of the problem of politics on both sides and we can easily see how extreme it can make people regardless of the thing being advocated.
Cherry-picking and self-reinforcement go hand-in-hand.
This deep desire to affect change in a controllable way...
This infinite desire for self value defined by external validation.
It's not sustainable. Perfection can only be obtained by observation of perfection of combined self through self and other.
It's okay to discard parts of yourself to balance yourself with your counterpart. A willing violation is no longer a violation.
Not observation of one or the other on a pedestal, but accepting that both are vital parts to the system and observing the perfection that comes from co-iteration.
Essentially turning a binary system quantum.