Next I want to explore the /api/v1/instance/peers endpoint.
Before I start with my own data collection, let's take a look at the data @Gargron published half a year ago (https://gist.github.com/Gargron/48e67b1b14723cd178c951fe7f373a38).
pawoo.net and mastodon.social were clearly the most connected instances.
On average each instance was connected to 175 other ones.
Fediverse as of 2018-08-24.
7 788 instances with 1 303 382 connections.
Let's look at the different clusters. They aren't clearly separated, but still visible.
On the left in orange is the English and European cluster. 43%.
I don't see any subclusters.
At the top in violet are several niche instances. I would say the only thing they have in common is the non-technical focus. 17%.
Finally in green a little cluster (4%) of instances with domains ending in m.to. Again mostly Japanese.
If you know more, please let me know.
Searchable (ctrl+f) SVG without lines to keep the filesize reasonable: https://lucahammer.at/vis/fediverse/2018-08-24-fediverse-min.svg
If you compare it with the screenshots, keep in mind that it is mirrored.
Beware of over interpretation. This is no scientific work. It's just one way to visualize instances. There are many others and probably better ones.
Qualifying the connections by amount of follow connections could draw a different image.
In the end it's not important for your everyday use. You can follow every account of every instance (as long as your instance doesn't cut ties with the other one).
Most important are the people you follow and what you toot.
I tried to clean up the script, but it's still messy. Feedback welcome: https://gist.github.com/lucahammer/eb68fc43a34877a304e7b81e5f729c4b
Data as GDF from yesterdays visualization (zipped, 10MB): https://lucahammer.at/vis/fediverse/2018-08-24-fediverse-GDF.zip
@Luca This is brilliant work. Thank you very much for doing this.
@Luca People, there's work to do! We're in the middle of the "niche instances" with a "non-technical focus"... 😟
@hiemstra It's easily possible that my quick interpretations are wrong. Niche is a rather broad term. And I would say it's a good thing.
@Luca I think it's pretty accurate (but we're supposed to be a technical university, so...)
Nice work, BTW!
@hiemstra Thanks.
Academia will probably always be niche.
it's cool how many instances there are that i've never even heard of in the violet area
@Luca Oh, Eldritch Café is actually quite visible on the violet cluster!
@Luca How did you differentiate between the violet cluster and the others? I can't believe all the instances in the orange cluster are tech focused for example.
@Luca Cool. Now can anyone tell us what they're each about? It's really hard to find an instance that matches individual interests!
@darwinwoodka I didn't discover topical clusters and I believe it doesn't matter that much which instance you choose. I believe trust into the admins is more important than the theme.
@Luca have you ever ran a plot of instance age (if you save that data) vs nb connections (maybe relative nb connections)? curious to see how they‘re related
@halcy I only have data from those two dates. I fear that would make the plot useless/misleading. Maybe someone else has more data. I think https://mnm.social/ collected those numbers.
@Luca too bad. And instances don‘t report age :(
@Luca what? mst3k is American!
@Elizafox English as in language.
@Luca Oh I misunderstood, heh. Wonder why we got pushed to the right a bit. Might be cause I sometimes speak French.
@scarlett The SVG is searchable (Ctrl+f). https://lucahammer.at/vis/fediverse/2018-08-24-fediverse-min.svg
@Luca wow, I can actually see my self-run single-user instance from here! I feel, quite literally, extremely seen. 😁
@Luca tag yourself
@Cdespinosa @Luca This looks amazing. Will there be a possibility at some point to drill down or find a particular instance?
@fuzzface @Cdespinosa I created a SVG that can be search through (ctrl+f): https://lucahammer.at/vis/fediverse/2018-08-24-fediverse-min.svg (Sorry for flipped axis)
@Luca 😮 This is a cool visualization. Liking this place more and more by the day.
@Luca do you have a hi-res version of this available by any chance?
@djsundog Only 5000x5000px at the moment: https://i.imgur.com/kexSunA.jpg.
Will look into rendering something bigger tomorrow.
@Luca Could you consider adding image descriptions to these posts?
@maloki Thanks for reminding me.
The images show names of instances that are connected through lines. They are one big hairball with two slightly visible groups of instances.
@Luca
Hat der 'Ball' oder wie man dieses Ding nennen möchte, wirklich leichte Ähnlichkeiten mit dem Icon von GoogleChrome? 🧐
Find die Form interessant. Alles schön Rund. Nicht so eckig und Kantig wie die von Twitter. 🤣
@Luca
Did you us umap or tSNE? Or something else entirely?
@vanderZwan #Gephi with ForceAtlas2 for the positions and Modularity for cluster detection.
@Luca
I admit I'm not too familiar with Gephi, but your description sounds like a "regular" force directed graph. If that already produces this, I'm curious what tSNE or umap would produce!
I bet even just feeding a 7788x7788 matrix with binary values based on whether they are connected or not would work well, it's kind of crazy how much structure those two algorithms can reveal!
@vanderZwan Which tools / libraries would you recommend? Preferably Python.
I will clean up my code and share the collection script as well as the collected data. Maybe you can use it.
@Luca
Well, there is only one real umap implementation at the moment, by Leland McInnes, who the guy who developed the algorithm. Luckily its in Python :)
https://github.com/lmcinnes/umap
Here he gives a talk explaining how it works in accessible terms at SciPy 2018:
https://www.youtube.com/watch?v=nq6iPZVUxZU
@Luca @vanderZwan Hey, I would seriously fancy a copy of the data to try some t-SNE dimension reduction.
@le_ArthurDent @vanderZwan Just published the script and data: https://vis.social/web/statuses/100610249989818510
@Luca @vanderZwan It appears that I have a hard time trying to wrangle this up into a matrix of connections, especially since I can't import gdf files directly into R. What is the node list telling me? Edge data seems to be the metadata of the instances.... but still there arent as many levels in the edge list as there are levels of nodes...
@le_ArthurDent @vanderZwan Yes, nodes is simple the data the instances provide about themselves. Only mastodon instances at the moment. Edges are what the Mastodon instances report as their peers. For each peer there is an edge (source, target).
If you tell me in what format you need the data, I could try to adapt the script.
@Luca @vanderZwan
I guess I got it. I was just wondering why the number of different instances in node1 varies from those in node2. How many instances are there? I assumed that there are 7788 instances having connection. So if I got the data right you have here a principal component analysis and a t-SNE of the connections. Colours are groupings obtained by k-means clustering, boldly guessing 10 clusters. I have too look further to map this for a better inspection of the instance names...
@le_ArthurDent @vanderZwan Not all instances are alive and/or providing information through the mastodon API. I was lazy with having a full list of nodes because Gephi auto-creates missing nodes from the edges list.
@Luca @vanderZwan Ahh... good righ, that implies that I might got the graphs right 😋
@Luca @vanderZwan I made some huuuuge plots with instance names for a visual insepction of the clustering. Just look how perfectly pixelfed.social stands out at the bottom of the scores plot. And in the t-SNE plot you see on the far right peertube and its associated servers *_*.
(I hope this is still readable after uploading it xD"...)
@le_ArthurDent
Nice!
(I love how the Internet leads to spontaneous collaborations like this)
@Luca
@Luca
Wow looks interesting, can You toot me an URL, that I can see it later on a bigger screen. On my 5 inch Xperia can't see so much ;)
On the right side in turquoise is the predominantly Japanese cluster. About 35% of the instances.