Since many people can't access the post anyhow, the short version of 'why' is: instagram keeps trying to block bibliogram (we can't have nice things), the author is tired of working around it, the author is also tired of fighting on a second front against the bots that try to scrape bibliogram. The bots also try hard to circumvent blocks that the author needed to put in place, rather than spending the time learning how to run their own instance and going to town on that.
Of course, it's open source so it's not like it's "discontinued" in the same sense as if gmail were to be announced to be discontinued. The official server will even remain online, just with current limitations (like viewing profiles being blocked by facebook) not being fixed for the foreseeable future.
> The bots also try hard to circumvent blocks that the author needed to put in place, rather than spending the time learning how to run their own instance and going to town on that.
So a three-way arms race, with the author fighting a similar battle in parts against the bots that is being fought in the other direction with their bot and fb/insta.
The enemy of my enemy may still be a pain in the arse for me!
> Of course, it's open source so it's not like it's "discontinued"
Aye, if there is enough value in it for someone else (who has the time and relevant skills) they'll pick up the reigns directly (if the author agrees) or in a fork – as happened when youtube-dl stagnated a bit.
> The official server will even remain online
And there are other instances. Or you can spin up your own if you don't mind (or want to try fix) the existing limitations & future breakages.
Looks like there's a referrer filter for HN. Copy URL and open in a new tab, I guess?
My personal Bibliogram instance has been blocked for months. With Instagram blocked as part of my wider PiHole block on Facebook's domain and public Bibliogram instances shutting down soon, I guess I'm going to just ignore Instagram links from now on.
One argument would be: Because beyond a certain size your service becomes a public platfrom and you are no longer competing with other platforms through your features, but through the fact that you have an existing userbase. So there is A) public interest in accessing, archiving, displaying, etc. the things users do on your platform and B) forcing your service to become more open the bigger it becomes keeps the competition of services going.
So have other services, i.e. PornHub and the like, crossed your threshold of public service?
I've always viewed this argument as a very poor one. This essentially mandates that all very successful online enterprises become government arms. When does Instragram/Twitter/Facebook, once made a "public platform", become free of these new responsibilities? Is Twitter free to shut down tomorrow once they have public fiduciary responsibilities?
I'm all in favor of additional regulation to encourage/reward/enforce "openness", especially with respect to data originating from users (photos, ...). But saying these companies have any responsibility to the social good is pure foolishness.
If you read my comment again you will find that I did not communicate this as my argument on purpose. So please don't frame it as such. Discussing something without necessarily having to share that opinion is hopefully still possible.
Also: we are talking about Instagram, not about Pornhub — I guess the nature of the service would certainly play a role in that argument (e.g. think about messengers).
What I do think myself is that beyond a certain size platforms should have it harder, should be held to higher standards, should fulfill more requirements etc. Because naturally bigger platforms will have it easier than their smaller competitors and it is in our interest to balance this out. Whether APIs are part of this, I don't know. Haven't given it much thought.
It has dominant market share of photo-centric social platforms.
In some smaller communities, it is also has a monopoly on social media in general and instant messaging.
This is even more problematic, because such platforms have a positive feedback loop of user acquisition. Many users have no option but to use them, because all their contacts are using them.
The definition of a tiny specific monopoly within a niche is meaningless. If you had 30 different factors to describe exactly what each platform does then each of them would be a monopoly in themselves.
In any case Instagram is certainly not in a monopolistic position at the moment unless you try to twist the sense of words to fit a narrative.
All companies have a responsibility to the social good, it's not foolishness it's been the way things work since forever. Companies are responsible for their employees' well being, for their impact on the environment, and they have many responsibilities to the government, this is all enshrined in many many laws.
See also all the companies that came together to produce Covid equipment at cost at the beginning of the pandemic.
It's just taking "Right to data portability"[1] a step further. We already have the ability to get data out of a service, but it's generally a slow and cumbersome process (e.g. giant .zip file that takes a day to create). So the next logical step would to require this process to be as seamless as possible, meaning real time import/export of selected bits of information, which could be accomplished by providing an API.
Basically, if we ever want to get the Internet back into the users hand, we need to decouple the "service" and the "storage" parts. Companies shouldn't be allowed to hold user data hostage.
I don’t mind requiring open APIs but can we all agree that this is pretty much in direct opposition to a lot of the privacy concerns that people here talk about?
The more able I am to interact with an interface programmatically, the more easily I can exploit it and turn it into a giant database.
The privacy problem with Instagram isn't what people post on it (they want it to be seen publicly), it's what the disgusting company behind it collects in the background as you interact with it (who may be a non-Instagram user and just need to visit an Instagram link you've been sent).
People don't use Bibliogram (or other alternatives) to access unauthorized data (all it displays is the same data visible on the web or via the app), it's to be able to browse without being stalked by Facebook and avoid their dark patterns.
People might not want to be seen publicly by everyone, which is why they allow users to choose who to show their posts to. But it’s not all that hard to exploit this by creating a popular page that lots of people end up sharing some data with. With permissive enough APIs, you instantly have a dataset.
This is more or less how the Cambridge Analytica fiasco started. And yes, this was seen as a privacy problem.
> People might not want to be seen publicly by everyone
Private accounts (where you have to approve followers manually) exist and as far as I know nobody is calling for making those public. A hypothetical API would still require the credentials of an approved account before returning data of such accounts.
> But it’s not all that hard to exploit this by creating a popular page that lots of people end up sharing some data with
I don't understand what this has to do with an API? You can create a phishing page now.
> This is more or less how the Cambridge Analytica fiasco started. And yes, this was seen as a privacy problem.
Cambridge Analytica created a malicious website that requested access to people's accounts and they granted it. Not only is it on idiots that approved the OAuth consent form, but it doesn't seem like it's got anything to do with what the parent comment was proposing? As far as I know all he was calling for is to provide an API that returns the public data that you can already get via the official web UI or app.
Do keep in mind that the people that are calling for a default API here aren't the same people that would've complained about the privacy implications of said API.
This is a diverse Forum and people have various opinions.
> Do keep in mind that the people that are calling for a default API here aren't the same people that would've complained about the privacy implications of said API.
To be honest I think they would have. I just don’t think they’ve fully thought through the consequences. Worse, maybe they have thought about the consequences, but are ok with being outraged at both ends of things. Lots of people get mad and try and skirt any security/password protection measure, but also get just as mad when they inevitably get hacked because they’ve skirted all the security measures. But even after that still won’t use the security measures because of the “inconvenience”.
If everybody accesses Instagram from bibliogram, bypassing ads and data collection techniques, who’s going to pay for the service? I know that your answer will be “so close it”, but that’s hardly what the users of the platform want.
Have you tried clicking the Instagram logo at the top left lately? It should open a drop-down menu where you can pick “following”, though I’m not sure if this has rolled out to everyone yet.
lol, OP completely missed out on the biggest IG ratelimit bypass. You could just use IPv6 and make literally millions of requests per second from a single server with a /48. (Oh! And before that you could just spoof X-Forwarded-For)
I printed money using this to automatically take good usernames as they became available.
Knowing someone who ran a public bibliogram instance, using IPv6 and rotating through addresses of a cellular provider's /32 it was, at the barest minimum, possible, but really annoying as Facebook would lock down any /64 once it felt it had looked at too many profile pages, and the limits were just as tight, if not more so, as the v4 limits.
Yes, but you could easily defeat this limit by switching to a /48, which gives you 65536 /64 subnets. Of course, even bigger IPv6 allocations are easy to get.
You could just get an allocation from RIPE for example, they'll give you a /29 without any justification required. Pretty much all dedicated server hosts will have a plenty of IPv6 space they can give you for a few $ per month.
Tunnels would've been too slow for my purposes.
> Can you set external IP per request?
Yeah, of course. You can just use the IP_FREEBIND socket option.
I am guessing that the $2M is from selling the usernames? Are the buyers individuals who want a good username, or bot farms who just need a large number of accounts?
I've never had a day job, I've spent almost two decades funding my lifestyle with silly projects like this. Everything from large scale world of warcraft botting to instagram registrations.
Are you not worried about running into legal trouble with these things? It seems like a gray area.
Also, what platform do you use to sell these things? How do you know the buyer won’t just take the credentials and not send payment? Or how do they know you won’t take payment and refuse to hand over the credentials?
> Are you not worried about running into legal trouble with these things? It seems like a gray area.
No, not really. I get the occasional nastygram from Hogan Lovells, but just shrug them off. I don’t do anything interesting enough for them to bother litigating.
> Also, what platform do you use to sell these things?
There are many, swapd.co is a good one.
> Or how do they know you won’t take payment and refuse to hand over the credentials?
Reputation, there are also a plenty of reputable escrow services available.
Very cool. I’ll echo what others have said, it would be very cool to see some sort of how to or read an AMA, about the strategies that maybe no longer work (since obviously you wouldn’t want to reveal something that is still making you money). I’d definitely read a blog post about how you used to make WoW bots or automate creating accounts
for all of the requests to their api, it constructs a very long query string that is a basic fingerprint of your browser/os. then the query needs to be signed (or else the api returns a captcha), which is done by a blob of encrypted/obfuscated-beyond-recovery JS that uses a more comprehensive browser fingerprint to validate the query string and generate a token that is appended to the query. there's another required query param which involves an xhr to phone home so that presumably the fingerprints can be checked server-side. finally all of your fingerprint data is sent to their api where it lives happily ever after and you get some 15 second videos to watch
Reminder to anyone who might want to peruse or fork the source in the future: Sources still available at https://sr.ht/~cadence/bibliogram/sources but as with anything may become unavailable at any point.
Referrer filter, so I right clicked and "open link in new tab" but still referrer filter. Don't like that the browser sends that info with that option. That should be equivalent to copy/pasting the URL.
I ran into the private profile problem with bibliogram ALL THE TIME. Even tho I was friends with them on instagram. So I just stop using it. It was broken most of the time anyways. It was a fun 3rd party unoffical API frontend that I added to my list of other ones I could self host.
That amount of hostility to scrapping and 3rd party clients feels like that they want to be in control of the data not you.
Thank you cadence for all the hard work and inspiration. This wasn't for nothing. I would bet on a newleaf-style reboot (with gallery-dl, to tackle onto a bigger community) if i had to. I might try my hand at it at some point but i'm afraid i'm less dedicated than you are.
I found a really awesome "public instance" vps node that had a bunch of utils & anonymous public interfaces webapps (nitter,...) running on it for all.
They also had a list of things they dont support, with reasons why. Meta interfacing were listed (including to bibliogram), and typically with links to fire-bot takedown requests to kingdom come. No one else has anything like this. Very very unique company.
I self-hosted this for a while but found that I got blocked/severely limited as soon as I tried to do basically anything. I'm sure there were things I could have done to get around the limitations that very quickly got imposed, but it wasn't worth it to me. If you don't like Instagram's UI/tracking/privacy policy/closedness then really the only sustainable solution is to stop using it, in any guise.
Bibliogram was a great project but I think Instagram is too aggressive about trying to protect its walled garden. It's a losing battle. Easier to just realise that, for most people, Instagram doesn't add that much to your life anyway.
Both are trivially easy to self host a private instance and with privacy redirect or a similar extension they just became my default way of opening twitter/reddit links.
Take down some of the most egregious pages and disinformation? No, we (Facebook/Meta) will waste our time shutting down anything remotely challenging to our monstrosity we're trying to conquer the world with (see also: shutting down the NYU Ad Observatory, or any countless other examples)
I don't really see the point of appealing to morality or to what ought to happen. They're not going to relinquish power willingly. Few individual do, and for groups it's vanishingly rare.
"don't be monopolistic assholes" is in the constitution and laws of several countries, it should not be an appeal to morality. The real problem is that at any point anybody does a move constraining internet corps, they get mocked by everyone as "regulating things to death". Yes that's the goal. Their death, in their current form at least.
> "don't be monopolistic assholes" is in the constitution and laws of several countries,
No it's not. People who formulate laws are specific, unless they're either creating a law that is intended to be ignored, or if they're creating a law meant for selective enforcement.
You can check every constitution and law code all day, you're never going to find anything telling anyone not to impede the attempts of an application not to constantly scrape their webpage. This kind of talk is purposeless.
That might be true in the anglophone world, but plenty of other populations aren't hostile to the idea of regulating the internet, and I don't mean the totalitarian countries. The GDPR is a ray of hope in this regard.
What prompted me to mention incentives in the first place is that the GP's framing of the problem is a common narcotic. Online is/ought-type complaining reduces the potential for action since the emotional need for signaling is already fulfilled. It's a trap I've fallen into many times myself. In itself, it's more dangerous at scale than any public mockery of regulation.
> Before we start: If Bibliogram has been helpful to you, please consider making a donation!
Before asking me to consider a donation, consider cutting the silly referrer based moan about the the site that referred me to yours so I have to jump through a hoop to see the content.
A reason that is not explained, so I'm going to assume is something petty. There are a couple of instances of sites doing this (for HN and other referral sources) because people said things they didn't like about them (or just in general) on the platform.
Of course, it's open source so it's not like it's "discontinued" in the same sense as if gmail were to be announced to be discontinued. The official server will even remain online, just with current limitations (like viewing profiles being blocked by facebook) not being fixed for the foreseeable future.