My dad used to work as a "system programmer" in Poland, starting in late 70's. He has worked in a biggest (I think) government facility back then. He used to write IO code for a database system. Because of a ban on IP imports across the Iron Curtain, or maybe how expensive DB/2 was, rewriting from scratch was the only option to have something that actually worked. It was all about registering child births, deaths, marriages, etc. So whole that database system was written by just a handful of people in S/370 Assembler (I think) using obviously 3270 terminal.
I was in his office number of times. Journey usually was starting in the morning. We did not have a car so we were taking a tram, about 45 mins to get there. Then about 5 mins walk and we are saluted by fully armed soldier. Then my dad calls the captain, using a phone handed over by another soldier behind the desk, asking if he can enter facility with his son. Permission granted, we enter.
For me, that was a completely different and magical world behind that gate. Everything neat and tidy, very unlike the world outside. My dad was taking me to his room (no open spaces back then) shared with one maybe two colleagues. Each had a terminal and a phone on their desks. Room was full of documentation books, punch cards here and there, other strangely formatted papers. I was allowed to wander through the office, use lifts (very hi-techy compared to what I saw before) and so on. I was 10. A boy in communist country in a middle of such facility.
Every now and then my dad was taking me to the main DC. One of the places I could not enter on my own. For me, it was stunning. Risen floor (with cables running below), noisy climate control and a room, 40m x 40m maybe full of mixture of washing machine or refrigerator size devices. One corner of that place was behind a glass with people in white coats there. Considering the noise, probably a good idea. One, maybe two of them, were wandering around in that DC looking at all the flashy lights.
So my dad was walking me around, telling me what those devices were: that's the disk (10MB If I remember correctly), that's the tape backup, that's a printer (A3 printer, noisy and fast), and here it is, that's the main CPU. I think it was IBM, but cannot be sure. There was also this Odra system there, probably with 1Mhz CPU, being a top of the line back then. In our country.
Later on my dad got a PC from his employer, it was a 4.77Mhz 8086 PC with 20MB disk. I was allowed to use it. This was the machine that got me into programming. It was so slow and noncompetitive to Atari and Amiga that the only fun thing to do with it was to program it. That sealed the deal.
I think limiting factor might be not number of cores and outside of erl scope, that is eth card they used, network infrastructure, etc. Even Elixir could be something that impacts the tests.
Without the business logic (which was in django IIRC) and deployment details, obviously. Very outdated and some later patches might be missing. No one was interested, you see.
I'd be surprised if there were problems with network, and if there were, that should have been obvious in the metrics.
That's the thing. You can always hack something in C to prove there is a better way for a specific task. In the past I did things like that just for fun. But in the real world it does not work like that. You buy into things as a whole, accepting their pros and cons as a whole. If you need to hack - change your tools.
Please do not beat the strawman. And don't set him on fire. He's innocent.
I offered to beat whatever you've done by tweaking the Py3 stdlib. Not by writing a plain C implementation.
If you for some reason doubt that this old python thing is of the real world - let me disappoint you. It was done because nothing else could do those 100K rps back then. And it did the thing for five years, until the whole stack was ditched.
I think you’re misinterpreting the point of the article. It’s not gloating about how much they scale, or saying their particular tech beats other techs. It’s just explaining how to solve a specific scaling issue on a specific platform.
As an Elixir user who had to deal with high connections/s in the past, I found it interesting and useful. I use Elixir for reasons that have nothing to do with performance so a language comparison isn’t particularly interesting.
This article is quite good, especially part about bottleneck caused by single supervisor in ranch. However I have to say that title is a bit misleading because all of this has nothing to do with Elixir, it's all about Linux kernel and Erlang, cowboy and ranch are written in Erlang.
Having said that, I will add that I think it is good to have Elixir.
And here i'm happy with my 8k requests per second Elixir IOT solution.
I use long lived processes and had to come up with some magic to work around the supervisor behavior with high child counts, etc.
Roughly I randomly assign worked to a node in the cluster if they have not yet been assigned. (there is some logic tracking total nodes on cluster and max / target that influences the decision). I verify if the remote (or local) worker is alive or migrating by checking a fragmented process Registry and unique identifier via :rpc (because I do some recovery logic if it's offline and let the caller specify if messages should force a spawn or can be ignored if the process is offline) and then pad the call with some meta information for rerouting so the receiver can confirm it is the same process the message was initially sent to (since processes cycle so frequently that the initial process may have died and a new process may have spawned in the time it took to forward the message).
If the process has changed mid transit or the worker has been flagged for migration the message gets rerouted to the new destination+process. If a process does not yet exist a hash based on the worker type + id and available worker pools is used to select which of the auto generated but named and registered (WorkType.PoolSupervisor_123) worker supervisors spawns the child node.
It's a trip, and needs to be heavily documented. Starting from scratch i'd probably change some things and it probably needs some refinement later this year before the next batch of 250-500k devices get added to the network, but the costs per reporting device are fantastic with plenty of low hanging fruit for improving the cogs further so I'm happy.
Point taken and I am already looking at stressgrid, "millions of users" is definitely a selling point to me. It is actually quite hard to generate enough and correct traffic to stress test large distributed systems.
Stick with the :module notation. This makes it clear that you need to be in "erlang mode"...indexes starting at 1, [probably] charlists instead of binaries, and possibly weird (from an elixir programmer's point of view) argument order.
That's fine I think. Elixir is their primary language it seems and using Erlang bits from Elixir is pretty easy, which is nice. It's a great benefit to the BEAM VM ecosystem. I think Elixir is the newer language and people enjoy up-voting it more. They could have said something like using BEAM VM and Cowboy maybe...? But I don't think it's a great misrepresentation either way.
Erlang. Same here. Reading first few pages of a book describing principles of OTP (processes, share-nothing, messages, etc) was mind blowing. Company I worked for at the time (and I still do) decided to switch from Java to Erlang in middleware area. This decision seemed like a mixture of insanity and enlightment. Do you switch from one of the most popular languages in the world to something that most developers never heard of? Surely, exciting, but will it work? How do we hire new staff? After our R&D confirmed it was promising, me with couple of other developers were tasked with rewriting quite an expensive piece of middleware software that was unfortunately reaching its maximum capacity. We had no knowledge of how the software worked, we just knew its API. We were given time to learn erlang so we did. We all switched from eclipse to vim (some to emacs). After a bit of playing around with erlang we did our job in just 3 months. New app was much smaller and was easily capable of handling many more messages than the previous one. And it was written by erlang newbies! Then many more erlang apps we have created. It turned out to be a really good choice. Also the level of introspection you get out-of-the box with erlang is just amazing. I have never seen anything like this before.
Now I can compare Erlang to Java and it is really baffling how the heck Java took over the world. To do erlang I just need an editor with some plugins, ssh connection to linux with OTP installed and of course rebar3. To do Java I need 4GB of RAM to simply run an IDE with gazillion of plugins, maven to cater for thousands of dependencies for the simplest app and I need to know Spring, Hibernate, AOP, MVC and quite a chunk of other 26^3 3-letter abbreviations. No thanks.
I already asked about this in the parent post that refers to Erlang, but do you happen to have a write up by any chance, where you go into more details. I’m super interested! It would be really appreciated. (This is not the first time I hear people praising Erlang in comparison to popular OOP languages)
