How our system evolved in the 3 months it's running.

1. ffmuc: FFMEET running a non-profit
conference system
3 months later

2. awlnx
● Annika Wickert
● Senior Network Engineer @AS51324
● Twitter @awlnx
Krombel
● Matthias Kesler
● IT Consultant
● Twitter @kr0mbel
2
Who are we?

3. 3
Jitsi to close the social gap during corona
• An upcoming Freifunk Meeting was about to get canceled because of Corona
• In the past we used NextCloud talk but we had issues (max 6. users per
conference)
■ We needed something different for at least 20 - 30 people
■ jitsi seemed easy and straightforward to install
● “apt install jitsi-meet” => Done ¯_(ツ)_/¯

4. 4
Debian package wins over Docker
• We had some issues with docker-proxy in the past
• docker image didn’t look up to date
• With scaling in mind
■ Planned using dedicated hardware for Jitsi anyway
■ Debian packages are available

5. 5
First test - looks promising
• Testing with some people of Freifunk Munich
• Worked like a charm, with a handful of people
• The idea was born:
■ Maybe other people have the same problem?
■ Why not open it for the public?
■ Maybe teachers, healthcare workers, etc. need that, too?

6. 6
We need insights! Let’s monitor!
• How many users are on the platform?
• What’s the impact on the Hardware/VMs?
• Do we have bottlenecks? If so: Where?

7. 7
What we needed for success
• First servers got donated by individuals (max 4 cores)
■ Went well as long as we only had small conferences
• Schools started to use our infrastructure => cheap servers were not
enough anymore
→ Videobridges need powerful hardware and we may need many of them!
→ Looking for sponsors

8. 8
Scaling the infrastructure

9. 9
The project gains attention
• Interviews
• BR, DigitalCourage, Focus Online, Stimme, SZ, …
• diverse Listen

10. 10
Problems start - Prosody
• Change network_backend to “epoll” - No more 1024 connections as a limit

11. 11
Don’t tune only one part of the stack - nginx
• 502 all over the place
• Raise the number of nginx workers
• Raise max open files limit

12. 12
How do we update without user impact?
• We can upgrade the videobridges one by one
• We query the videobridge API and wait for 0 users then run our custom
upgrade script
• Upgrades of videobridges just happen in the background during the day/night
• Jicofo/Prosody upgrades … sadly we have to do them manually but we don’t
need to update them very often

13. 13
Holidays are over … school starts - 20.04.2020
• All time record before was around 600 Users

14. 14
21.04.2020 - Another record! Sadly also another bug
● 20 min. to find the issue, isolate and fix it!
● Unfortunately there was an other bug hiding after the crash
● We run at reduced capacity for a few hours but as we have so many servers
that was not an issue.
● Crash of prosody crashed some important DBs :/ ⇒ Loadbalancing failed

15. 15
New records every day!
• On 29.04.2020 we hit another highscore of 1503 concurrent users. Servers had
no issues!

16. 16
People tend to be in firewalled networks => TURN
• To connect devices which cannot properly transmit data to the videobridge
(UDP blocked)
• videobridge2 used to provide a TCP Fallback to 443 … but it’s not real
HTTPS traffic so NGFW tend to block it
• Solution TURN (Traversal Using Relays around NAT) RFC5766 and 6156
• Ugly “Feature” in browsers => They prefer the first turnserver in the list … =>
Patch Jitsi to randomize the sequence

17. 17
People ask for bigger meetings
• Octo solves the problem of only one videobridge per conference (server-side
is no longer an issue)
• Sadly video traffic is unencrypted between videobridges => we need a
solution for that
• As we do high packet rates we need a full-mesh VPN without a central point
where all traffic has to pass
=>

18. 18
82 people in one meeting and nebula in action

19. 19
Octo on all bridges

20. 20
Bugs with Octo
• Videobridges join XMPP via pubsub and MUC => no region info in pubsub
• Update to jitsi-videobridge2 unstable

21. 21
We are running on the internet aren’t we?
• From time to time there are issues in the internet beyond our control (route
leaks, connectivity issues between providers, maintenance)
• We have many servers and providers so we can easily mitigate connectivity
issues

22. 22
We got more popular so DDOS attacks happen
• People attacked our infrastructure and didn’t stop when they couldn’t bring
down #FFMEET
• They started to attack other parts of our infrastructure

23. 23
Custom code build on upstream/master
• We no longer run the upstream packages for most software in the stack
• We run a fork from latest master with some additions
• jitsi-videobridge2 is still the latest unstable package on all hosts
• We forked the electron app

24. 24
Our infrastructure 3 months ago!

25. 25
How it evolved

26. 26
Some figures? 3 Months of #FFMEET
• ~ 810TB videotraffic
• ~ 847.955.871 HTTPS Requests
• ~ 360 CPU Cores / 720GB of RAM
• ~ 800.000 Unique Users
• ~ 500 tickets solved (many schools, charity projects, individuals etc.)
• ~ 200 Euro / day
• 2 Admins and at least 15 people in the background for translations, press,
wiki etc

27. 27
Some unexpected things happen
• You get famous in Peru and they use
#FFMEET as their primary conf system
• Schools start to depend on you
• Small companies start to depend on
you

28. 28
What’s next?
• Hopefully more happy users!
• Establishing a long term roadmap for the service
• Gathering money and resources to secure the funding of the project even
more
• Every penny helps! paypal.me/ffmucspenden

29. 29
Thank you very much!
A special thanks to all who helped
writing/translating the FAQs.
Also thank you to all the users and of
course our supporters:

30. 30
Firefox breaks user experience
• Users experience problems especially in large conferences (25-40 people)
• Add warning for Firefox users … it’s mostly ignored … so problems still exists
• Deactivate Firefox support and release FreifunkMeet app
→https://github.com/freifunkMUC/jitsi-meet-electron/releases

31. 31
DoH / DoT Public Resolver
• As DNS is an unencrypted protocol there are approaches to develop a more
secure one
• We offer DoH (native in Firefox) / DoT (native since Android 9) / DNSCrypt