FOSDEM 2018
What is FOSDEM?
FOSDEM (Free and Open-source Software Developer’s European Meeting) is one of the largest European open-source and non-commercial conference, taking place annually at the Université Libre de Bruxelles. This year it offered more than 650 lectures presented by 600 speakers on 50 different themes. It’s a real place to share and learn about open-source.
In keeping with the ideals of open-source, FOSDEM is an open and free event, with no registration needed and no entry fees, which allows more than 4,000 people to attend each year.
It’s mostly funded by sponsors and sales of goodies during the weekend. Most organizers are passionate volunteers, I’ll let you imagine how many there are!
My Fosdem experience
I had the chance to attend the FOSDEM event for the first time this year, with 8 other Deezer folks from different teams (Core Infra, Core Data and Engineering Effectiveness).
The more, the merrier, right? It was exciting to attend the event, all the more when I saw all the many different tracks and lectures.
When we landed on Friday, we charged around Brussels’ city center on a whistlestop tour (some of us also tested the famous Belgian beers, for research purposes).
We decided to make an approximate draft schedule for our stay and bookmarked the presentations we wished to attend using the FOSDEM app. To make the most of the event we focused on several subjects we are working on or want to work on in Deezer infrastructure.
I set my priorities on subjects like containers and databases so mainly on lectures about Kubernetes and MYSQL. We also wanted to see some presentations on different tools that help us manage our Deezer infrastructure such as Prometheus monitoring, debug tools like strace, dtrace and Grafana for dashboards, etc.
On Saturday morning we started with the first meeting “Welcome to FOSDEM 2018” as an introduction in the biggest, fully packed amphitheater of the university. It was the beginning of a super two-day knowledge marathon surrounded by passionate people happy to share and learn.
I liked that feeling to be a student again in university amphitheaters with the rush to join the “class” before the bell rings and the importance of taking notes to remember the important information among all that you hear and see.
At FOSDEM everything goes fast. The format of the lectures is short, with most of them under 30 minutes so you don’t want to miss the start. It’s quite a nice way to attend many different lectures but it can also be a bit frustrating sometimes when the subject is very interesting or when you’re stuck in the middle of a crowd, trying to reach the room.
Fortunately we were able to attend most of the lectures we were interested in and gleaned lots of information on the subjects that are important for us to improve our infrastructure at Deezer.
Discovering Fosdem for Deezer
We were able to learn more details about certain tools (ProxySQL, MySQL InnoDB cluster, etc.) to help us with our upcoming study on our MYSQL databases fault tolerance and performance, which are very important aspects of our infrastructure at Deezer. We are secure with our actual processes on MYSQL (failover, replications, backups) but our challenge is to always improve our infrastructure, just like we did with our work on memcache fault tolerance with mcrouter.
Tools like ProxySQL could help us improve reliability by executing more graceful switchover and failover without breaking transactions as they understand the MYSQL protocol and will act accordingly, which other proxy layers like HaProxy can’t do. Another interesting feature in such tools are the dynamic query routing (for example, forwarding queries to specific Mysql backend based on the query type) and possibly the query caching part that could help better optimise databases performances.
We also went to lectures on Kubernetes, which is a big topic we started to work on with the project “Infrastructure as Music”. This powerful container orchestration tool is open-source and originally developed by Google.
It will ease our infrastructure management, as well as our developers’ work, and help operate more easily with working as microservices at Deezer. Kubernetes impacts different fields of our infrastructure (network, security, code deployment, dependency between dev and infra teams ) so learning about its various topics was very helpful to orientate our work.
Finally we attended lectures on many different tools we already use to manage our infrastructure on a daily basis. We learned about the new features offered by Grafana’s new version (5.0), which is now ready to be used by our different tech teams, and also on Prometheus monitoring.
At Deezer, Grafana dashboards are used by almost all the tech teams. It can be simple dashboards with server resources information but also application errors (sql, php, etc.), web monitoring (network traffic, memcache, mcrouter, number of connections and requests, adn/cdn, etc.), database monitoring (replication, slow queries, threads, etc.), specific content metrics (encoding, publication, fraud, streams), Kafka bus message metrics (consumers offset/lag, brokers metrics, topics, etc.) and many more.
We use different data sources such as Elasticsearch, InfluxDB but mostly Prometheus (v2.2), which is our main monitoring tool (working with consul and node-exporter). With Prometheus at Deezer, we currently have over 2.3 million time series, scraping around 60k samples per second and 300 servers are monitored.
You can refer to the meetup we held on Prometheus with Antoine Leroyer to get more details (video & slides).
These tools among others allow our tech teams to have an overview of Deezer’s infrastructure and services and to be proactive when necessary.
FOSDEM 2018 was a rich weekend! I can’t talk in details about all the topics we covered and all that we shared or learned but it was a really great experience to live with my teammates (Cédric, Antoine, Luc, Alexis, André, Alexis, Benjamin and Romain) and I recommend it.
A good surprise also was the meeting about the Unix architecture evolution from 1970 to 2018, which I didn’t plan to attend at first. It was an amazing reminder of all the work done on its architecture and how it all started.
The organisation behind events like FOSDEM must be quite heavy and we thank all their teams for the hard work.
Some tips for FOSDEM 2019
If you wish to attend the FOSDEM, preparation is key!
The place is huge and composed of many different buildings. All the lectures you are interested in will probably not take place in the same building and it can take up to 10 mins sometimes to reach a room. To avoid the disappointment of missing some lectures, try to plan your schedule accordingly. Given the huge crowd, it also happens that the room is full and you can’t attend.
The official website contains a lot of information to help you and an application is also available. A nice feature in the app is the ability to bookmark each lecture you may be interested in, so that you can view your own schedule for the two days in the “Bookmarks” tab.
And, most of all, be prepared to learn a lot, enjoy yourself and don’t miss the belgian fries and waffles, you can’t leave without tasting them!
