I have done that at meta/FAIR and it is published in the Llama 3 paper.
You usually start from a seed. It can be a randomly picked piece of website/code/image/table of contents/user generated data, and you prompt the model to generate data related to that seed.
After, you also need to pass the generated data through a series of verifiers to ensure quality.
For the knowledge preservation, I guess that a copy of deepseek has most of the required information. But, it would be hard to run it in a primitive world.
Thanks. I think the knowledge should contain at least text, image and tutorial videos. It would be nice if it contains some VR training programs too.
The only problem is, we need to find secure areas and long term power supply for these stuffs. The "good" thing about the virus is that it's not going to kill the infrastructure, but it kills the people who maintain them.
I'm thinking, maybe private citizens should prepare such a small library as well, just for survival. There must be some VR games about camping, fishing, tool making, etc.
But we have landfills which are full of great raw materials. I would argue that it is easier to collect steel from a landfill than from a mine during the industrial revolution.
Collect, yes. But those materials, at the end of the consumer use cycle, are definitionally about as far as possible from raw. Having collected this scrap steel, how do you propose to smelt it efficiently for reuse with little or nothing that burns hotter than the local hardwoods? That's the question actually being asked.
I believe you can work some grades of steel tolerably with manual, bellows-fired forge processes, but not all of it will still be the same kind of steel when you finish (decarburization, etc), some you won't be able to meaningfully hot work at all, and you likely won't be reliably able to produce pieces much beyond the quality you could get in bloomery days - forgings and hot working would benefit from well-chosen scrap of compatible metallurgy, but everything would tend over time to rehomogenization into something between wrought iron, and what we would now call low- to medium-carbon mild steel.
That's far from nothing, you can do at least as much with it as our ancestors did, but it also isn't close to anything we'd call "modern." Between the relatively
enormous energy inputs required to do any meaningful hot working and the relative scarcity of materials no longer being manufactured, tools and objects made of iron would probably come to be family heirlooms again for more than sentimental reasons: replacing your hipster thrice-great-grandfather's cast iron might indebt you the equivalent of half a year of your struggling truck farm's proceeds.
(It is still called a 'truck farm.' No one knows why. The old missus in town who reads says it has to do with some of the old machines, but even she doesn't try to pretend she ever saw one of them move, so no one thinks much of that. But all the village, not only the half or so she's midwifed, is happy to grant her her modest notions.)
"Not raw" is actually generally an advantage, as it's possible to sort through landfills to find directly-usable materials, or those which can be fed to an electric arc furnace (presuming that level of sophistication), or a charcoal-fed blast furnace.
This isn't great, mind you, but it's a good start.
Steel has a relatively high melting point. Other metals, notably aluminium and copper can be worked at far lower, far more attainable temperatures.
Keep in mind that modern steelmaking really doesn't begin until the Bessemer process (1860s), and that was far more predicated on high-volume, high-quality fuels (anthracite coal) than it was in the input iron ore grade. Knowledge of and access to liquid oxygen, far better process (and temperature) controls, and improved metallurgy, through the 19th, 20th, and 21st centuries have advanced smelting and fabrication even further.
NB: "Truck" means to barter or trade. A "truck farm" is one on which cash crops (rather than those for local consumption) are grown, usually vegetables rather than staple grains (wheat, maize, rice).
I asked no question for you to answer, save the one you actively and at not obviously needful length failed to treat at all. How do you propose to power an electric arc furnace on peat, coal, or hardwood? How do you propose to produce liquid oxygen with like premodern fuels, which the problem statement declares as a constraint? Save the trivial point that less useful metals with lower melting points are easier to work, what does any of what you've said have to do with anything I and my prior interlocutor actually were conversing about?
That last question, though at least actually asked, also requires no response. Its answer is obvious, also trivial, and unrelievedly in the negative.
I suspect you and I share similar pessimistic views of what a post-industrial, post-carbon, post-collapse scenario might be capable of. I was amending and supplying technical corrections to your points, though I agree generally that landfills won't come close to replacing historical raw material sources.
Addressing the question of metalworking and presuming a rubbish tip source, it's useful to note:
- Previously refined metals won't need refining or smelting but rather recycling. This generally simply requires heat. Aluminium does not, for example, require (electrically-driven) reduction, iron and steel do not require (coke, oxygen, or hydrogen based) carburation. The heat requirements remain challenging, but it is still a lesser challenge than production from ore.
- There are conceivable thermal processes which might assist, and for which technological knowledge even in the absence of prodigious energy resources might suffice. Solar thermal energy (requiring polished mirrors, but these being reasonably attainable) can achieve temperatures of 3,500 °C (6,330 °F). Steelmaking "only" requires temperatures of ~1,700 °C. Total capacity of a solar furnace would be well below that of a fuel- or electrically-powered blast or arc furnace, but useful quantities of metals and glass could likely be produced without extirpation of forests for charcoal.
- Siting near geothermal or hydroelectric resources, and presuming electrical generation, could enable electric-arc furnaces. Even today much aluminium production is opportunistically sited near such cheap power sources.
Several of these options might not be immediately available following a widespread collapse, but could be bootstrapped within reason over time, though most likely at far lesser scales than at present.
My suspicion is that a post-collapse society, and/or a future technological society operating with a rewewable energy basis (biomass, hydroelectric, geothermal, solar, wind) would probably have very different material bases (far more stone, brick, and ceramic, some plant-based materials whether structural timber or plant-fibre-reinforced ceramics), a vastly different land-use pattern (concentrated rather than sprawled settlements) and transportation (water-borne, canals, heavy reliance on pedestrian travel, possibly electric-powered transit and freight, minimal air travel). Overall energy-intensity comparable to the late 19th / early 20th century in the US and Europe may be reasonably attainable with smaller populations, and on balance that wasn't particularly burdensome. Food production is probably the major hurdle without Haber-Bosch ammonia production. Sufficient farm mechanisation given some available fuel and/or power distribution (alcohol, biodiesel, possibly synthetic hydrocarbon production, elecricity) would have a huge societal benefit even at small fractions of present total and per capita energy usage, not just for agriculture but other high-benefit uses such as marine propulsion and remote heat and power.
This is a discussion site, and the discussions occur between more than just two people (e.g., comment and response). I was responding with additional context for any reader interested, though of course you can be presumed to have interest. Discussions are not however proprietary, and shouldn't be treated as such.
Come on... Meta has been refining pytorch for more than a decade. It basically contains all that you need to train LLMs, including the latest technologies. What more do you need? The part of the code that is specific to Meta infrastructure?
The reasoning happens in the chain of thoughts. But OpenAI (aka ClosedAI) doesn't show this part when you use the o1 model, whether through the API or chat. They hide it to prevent distillation. Deepseek, though, has come up with something new.
Training a 1B model on 1T tokens is cheaper than people might think.
A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model.
So you would need around 4,400 hours of GPU training costing only $11k
And costs will keep going down.
You require 6 * parameter * token flops[1] to train LLM. Which means (flop/s of H100 * MFU) / (6 * parameter) token per second. Assuming MFU of 40%, it is (1000 * 10^12 * 0.4) / (6 * 10^9) token/sec = 67,000 token/sec.
This repo[2] by Meta achieves 48% MFU, or 80k token/second.
But, if the non-profit gives all its assets to the new legal entity, shouldn't the new legal entity be taxed heavily? The gift tax rate goes up to 40% in the US. And 40% of the value of openAI is huge.
Except that a plane has passengers. But this rocket had none. It did not even have cargo. And it crashed in a pre-evacuated zone. There is no need to have the same level of security for these two situations.
He doesn’t need to be a vet to know the difference between a dog and a cat. Retrieving the booster is optional. Boeing, their competitor, can’t even do it.
So because Boeing can't do it, we should just forget about safety investigations and let SpaceX do whatever? That logic doesn't fly. Neither does your nonsense analogy. Either we give a shit about safety or we don't. FAA previously grounded the Falcon 9 and cleared it to fly once they determined it was safe. They will do the same here. I feel like you and others are severely misjudging the formalities and expertise required for these things and so you're just armchairing this shit. It's tiring. You're not as smart as you think you are.
Yeah because Boeing can't do it and the FAA is OK with it, then SpaceX should be held to THAT same standard and not judged differently otherwise it treates SpaceX differently and contributes to complaints of political double standards. If it's safe enough for a Boeing booster to burn up on entry then the line should be drawn there. If SpaceX managed to land a booster to help recover costs that's a financial benefit to them and has no impact whatsoever on safety.
