sahilister @sahilister@fosstodon.org

How long do you think “awhile” lasts?

A little bit. And then some.

— Ricardo Siri https://twitter.com/linierscartoon/status/1443422662760488966?s=20

#Quote #mjbQuote #Life .

We have an Open Collective at
https://opencollective.com/bifrost-mam

Contribute to this fund raising campaign so we can fund two developers to work full time on this project for one month.

Spread the word so more people know about this campaign.

#Matrix #XMPP #FreeSoftware #OpenSource

We found a missing feature in a Free Software we use and we are taking advantage of the freedom to modify the software. We need your help to make it happen.

#Matrix and #XMPP are two federated/decentralized instant messaging protocols. Both has its pros and cons. #Bifrost bridge is an effort to connect these two protocols. Currently the bridge does not support offline messages for XMPP users, making it unreliable for XMPP users.

The bridge being #FreeSoftware we want to fix this problem by adding support for offline messages to the bridge.

Join our meeting on 4 Nov 2021 meeting 6 PM IST / 12:30 PM GMT https://meet.fsci.in/bifrostmam if you like to help.

Quick heads-up: The Free Software Community of India let us know about their Software Freedom Camp 2021, an online mentorship programme. Last date to register is 5 November: https://camp.fsci.in

#inPURRection #gattini #kittens

The recently completed Linux Plumbers Conference (LPC) 2021 used the Big Blue Button (BBB) project again as its audio/video online conferencing platform and Matrix for IM and chat. Why we chose BBB has been discussed previously. However this year we replaced RocketChat with Matrix to achieve federation, allowing non-registered conference attendees to join the chat. Also, based on feedback from our attendees, we endeavored to replace the BBB chat window with a Matrix one so anyone could see and participate in one contemporaneous chat stream within BBB and beyond. This enabled chat to be available before, during and after each session.

One thing that emerged from our initial disaster with Matrix on the first day is that we failed to learn from the experiences of other open source conferences (i.e. FOSDEM, which used Matrix and ran into the same problems). So, an object of this post is to document for posterity what we did and how to repeat it.

Integrating Matrix Chat into BBB

Most of this integration was done by Guy Lunardi.

It turns out that Chat is fairly deeply embedded into BBB, so replacing the existing chat module is hard. Fortunately, BBB also contains an embedded etherpad which is simply produced via an iFrame redirection. So what we did is to disable the BBB chat panel and replace it with a new iFrame based component that opened an embedded Matrix chat client. The client we chose was riot-embedded, which is a relatively recent project but seemed to work reasonably well. The final problem was to pass through user credentials. Up until three days before the conference, we had been happy with the embedded Matrix client simply creating a one-time numbered guest account every time it was opened, but we worried about this being a security risk and so implemented pass through login credentials at the last minute (life’s no fun unless you live dangerously).

Our custom front end for BBB (lpcfe) was created last year by Jon Corbet. It uses a fairly simple email/registration confirmation code for username/password via LDAP. The lpcfe front end Jon created is here git://git.lwn.net/lpcfe.git; it manages the whole of the conference log in process and presents the current and future sessions (with join buttons) according to the timezone of the browser viewing it.

The credentials are passed through directly using extra parameters to BBB (see commit fc3976e “Pass email and regcode through to BBB”). We eventually passed these through using a GET request. Obviously if we were using a secret password, this would be a problem, but since the password was a registration code handed out by a third party, it’s acceptable. I imagine if anyone wishes to take this work forward, add native Matrix device/session support in riot-embedded would be better.

The main change to get this working in riot-embedded is here, and the supporting patch to BBB is here.

Note that the Matrix room ID used by the client was added as an extra parameter to the flat text file that drives the conference track layout of lpcfe. All Matrix rooms were created as public (and published) so anyone going to our :lpc.events matrix domain could see and join them.

Setting up Matrix for the Conference

We used the matrix-synapse server and did a standard python venv pip install on Ubuntu of the latest tag. We created around 30+ public rooms: one for each Microconference and track of the conference and some admin and hallway rooms. We used LDAP to feed the authentication portion of lpcfe/Matrix, but we had a problem using email addresses since the standard matrix user name cannot have an ‘@’ symbol in it. Eventually we opted to transform everyone’s email to a matrix compatible form simply by replacing the ‘@’ with a ‘.’, which is why everyone in our conference appeared with ridiculously long matrix user names like @jejb.ibm.com:lpc.events

This ‘@’ to ‘.’ transformation was a huge source of problems due to the unwillingness of engineers to read instructions, so if we do this over again, we’ll do the transformation silently in the login javascript of our Matrix web client. (we did this in riot-embedded but ran out of time to do it in Element web as well).

Because we used LDAP, the actual matrix account for each user was created the first time they log into our server, so we chose at this point to use auto-join to add everyone to the 30+ LPC Matrix rooms we’d already created. This turned out to be a huge problem.

Testing our Matrix and BBB integration

We tried to organize a “Town Hall” event where we invited lots of people to test out the infrastructure we’d be using for the conference. Because we wanted this to be open, we couldn’t use the pre-registration/LDAP authentication infrastructure so Jon quickly implemented a guest mode (and we didn’t auto join anyone to any Matrix rooms other than the townhall chat).

In the end we got about 220 users to test during which time the Matrix and BBB infrastructure behaved quite well. Based on this test, we chose a 2 vCPU Linode VM for our Matrix server.

What happened on the Day

Come the Monday of the conference, the first problem we ran into was procrastination: the conference registered about 1,000 attendees, of whom, about 500 tried to log on about 5 minutes prior to the first session. Since accounts were created and rooms joined upon the first login, this is clearly a huge thundering herd problem of our own making … oops. The Matrix server itself shot up to 100% CPU on the python synapse process and simply stayed there, adding new users at a rate of about one every 30 seconds. All the chat tabs froze because logins were taking ages as well. The first thing we did was to scale the server up to a 16 CPU bare metal system, but that didn’t help because synapse is single threaded … all we got was the matrix synapse python process running at 100% one one of the CPUs, still taking 30 seconds per first log in.

Fixing the First Day problems

The first thing we realized is we had to multi-thread the synapse server. This is well known but the issue is also quite well hidden deep in the Matrix documents. It also happens that the Matrix documents are slightly incomplete. The first scaling attempt we tried: simply adding 16 generic worker apps to scale across all our physical CPUs failed because the Matrix server stopped federating and then the database crashed with “FATAL: remaining connection slots are reserved for non-replication superuser connections”.

Fixing the connection problem (alter system set max_connections = 1000;) triggered a shared memory too small issue which was eventually fixed by bumping the shared buffer segment to 8GB (alter system set shared_buffers=1024000;). I suspect these parameters were way too large, but the Linode we were on had 32GB of main memory, so fine tuning in this emergency didn’t seem a good use of time.

Fixing the worker problem was way more complex. The way Matrix works, you have to use a haproxy to redirect incoming connections to individual workers and you have to ensure that the same worker always services the same transaction (which you achieve by hashing on IP address). We got a lot of advice from FOSDEM on this aspect, but in the end, instead of using an external haproxy, we went for the built in forward proxy load balancing in nginx. The federation problem seems to be that Matrix simply doesn’t work without a federation sender. In the end, we created 15 generic workers and one each of media server, frontend server and federation sender.

Our configuration files are

systemd target file
systemd generic worker (yaml)
systemd federation sender (yaml)
systemd frontend (yaml)
systemd media server (yaml)
nginx file

once you have all the units enabled in systemd, you can then simply do systemctl start/stop matrix-synapse.target

Finally, to fix the thundering herd problem (for people who hadn’t already logged in), we ran through the entire spreadsheet of email/confirmation numbers doing an automatic login using the user management API on the server itself. At this point we had about half the accounts auto created, so this script created the rest.

emaillist=lpc2021-all-attendees.txt IFS=' 'while read first last confirmation email; do bbblogin=${email/+*@/@} matrixlogin=${bbblogin/@/.} curl -XPOST -d '{"type":"m.login.password", "user":"'${matrixlogin}'", "password":"'${confirmation}'"}' "http://localhost:8008/_matrix/client/r0/login" sleep 1 done < ${emaillist}

The lpc2021-all-attendees.txt is a tab separated text file used to drive the mass mailings to plumbers attendees, but we adapted it to log everyone in to the matrix server.

Conclusion

With the above modifications, the matrix server on a Dedicated 32GB (16 cores) Linode ran smoothly for the rest of the conference. The peak load got to 17 and the peak total CPU usage never got above 70%. Finally, the peak memory usage was around 16GB including cache (so the server was a bit over provisioned).

In the end, 878 of the 944 registered attendees logged into our BBB servers at one time or another and we got a further 100 external matrix users (who may or may not also have had a conference account).

https://blog.hansenpartnership.com/linux-plumbers-conference-matrix-and-bbb-integration/

We now have the rough schedule published at camp.fsci.in/sched/ We will share more details about each phase in the coming days as soon as we finalize those details.

Feel free to contact us anytime you have any questions.

#sfcamp #freesoftware

Planets of Free Software Community

Knowing our own community is important.

GNU https://planet.gnu.org
Debian https://planet.debian.org
Ubuntu https://planet.ubuntu.com
LibreOffice https://planet.documentfoundation.org
KDE https://planet.kde.org
GNOME https://planet.gnome.org
Firefox https://planet.mozilla.org
Emacs https://planet.emacslife.com
PHP https://www.planet-php.org
Python https://planetpython.org

#FreeSoftware #OpenSource

Yes it's an hour. And it's totally worth it. All of it.

I seriously contemplated not adding any quotes, because you should see the whole talk to fully understand the context as #ContextMatters which also makes the jokes better then they already are.
If only it weren't so true at the same time.

I didn't quote a line from that talk that stuck in my head ever since I first heard this talk, which likely was in 2010.

This is still one of my all time favorite talks:
Eben Moglen - Freedom in the Cloud (2010):
https://archive.org/details/EbenMoglen-FreedomInTheCloud2010/

Eben Moglen is an incredibly gifted story teller with an impeccable analyses and an inspiring message.
And funny too.

"[FB is] *grossly* overpriced at spying-all-the-time. ... I'm not suggesting it should be illegal; It should be obsolete. We're technologists. We should fix it."

"Mr. Zuckerberg richly deserves bankruptcy. Let's give it to him.
For Free."

#FreedomBox

We're glad to announce that the registration for both learners and mentors are open for Software Freedom Camp Diversity Edition - camp.fsci.in/news/sfcamp-dived…

Apply as learner - camp.fsci.in/learners/
Apply as mentor - camp.fsci.in/mentors/

Poster credits - @avronr and @kannan

#sfcamp #freesoftware
Software Freedom Camp 2021

We're now live on from our new account on Fediverse at @sfcamp

Now you can follow us from Friendica, Mastodon, Pleroma, Diaspora, Hubzilla, GnuSocial, Misskey or amongst other places.

#sfcamp #freesoftware

You are invited to meet your daily online meeting needs with the community-run and free software powered FSCI Jitsi service here https://meet.fsci.in.
For donations please check out https://fund.fsci.in
#FreeSoftware #videoconferencing

What kind of world are you helping to create? Is it the world you want to live in yourself? That you want your children to grow up in?

If not, it might be time to start doing something else.