The last weekend on Earth

Hi there, we had a hellish weekend on WikiToLearn and now I will talk about what happened the last two days. Let’s start from saturday.

Saturday, April 09

In the last two months, tech developers, helped by students from India in order to the Google Summer of Code, worked hard, together with the editing team, on new features and fuctions that we wanted to implement in the site. So, tech developers decided to release WikiToLearn 0.7 on saturday April 09. But something went wrong, and what would have to be an ordinary saturday of releasing became a hellish saturday.

The meeting on hangout was set at 09:00 AM GMT+1. Developers met at that hours in hangout and started working on the release. But something, from the beginning, didn’t want to go well: our instance on Bari was not ready to work. Our new system is hosted in Bari, near to the GARR mirror, but for som reasons it was not ready to work. Finally, thanks to the great job Paolo did, all was ready to go, and they started.

The new WikiToLearn is online in the GARR network, and this wass really helpfull for releasing: guys could download every single file of the open-source software (ubuntu, debian, manjaro) without leaving the building, so the release would have to be easy. But if something starts bad, it will keep going bad.

They had a BIG dataloss. How much big? As the entire home page. A complete disaster. They were not be able to use their back-ups to restore all the database, so it took more time they expected. In the afternoon, at 05:00 PM I joined the hangout, and the situation was not so happy. I found Dario and Luca near to desperation, working hard to restore the database and to release the new site. I am an editor, I on’t know anything of coding or programming, but I know well how the site works, so I rolled up my sleeves and helped them in 100 minutes of full debugging: comparing the production site with the local istance I could say them what worked well or bad, so they could fastly repair; my help was nothing compared with all the work they did, but I am happy about having helped them in that moment.

Obviously, the job took another extra-time. In the night they were still working, FROM THE MORNING! They stayed over 13 hours at pc working on the release but, at the end of the day, we were online. It was 10:21 PM when Riccardo finally announced us: WE ARE ONLINE. RELEASE DONE.

For this hard work, I want to thank all the release team: Luca, Dario, Paolo, Davide, Gianluca, Cristian and the others who spent a saturday on a pc for release. Congratulations guys, you are awesome.

Sunday, April 10

If you thought everything was finally done, you are wrong. Releasing WikiToLearn 0.7 was really hard, but in the morning of Sunday we had a bad new: we were under a spam-attack. It was awful: cloudflare died, someone passed the captcha and a HUGE number of new users, created by bots, started creating new spam pages on the site. Since the morning.

When I woke up, after having read the messages, I immediately turned on my pc and joined Luca in hangout: we needed an anti-spam filter. Riccardo and Davide was cancelling the spam pages and blocking the fictitious users, I helped them. Then, as I did in saturday, I helped Luca in the debugging fase: he instantly installed the anti-spam filter and I immediately tried it. Obviousy, it didn’t work from the beginning. Good, another nightmare. What a hellish weekend, damn!

For some reasons I had to stop my work and I couldn’t help them, due to my not being at home. I followed the processing from mobile, doing some little debugging, but it was really awful. In the early afternoon we decided to stop the registering process by making an impossible captcha. It was a drastic decision, but it was necessary. The guys worked hard all the evening, and, at the end, they finally managed to install the anti-spam filter and to block the spam-attack. After the awful saturday, they did not deserve to spend another day on their pcs to work on the site. But they did it, because they really believe in this project.

After this weekend, I really want to thank all the guys who hard worked on the site. Thank you guys, I love you. You make people like me able to write and to use the site, so we are very thanksfull.

Is it all past? Obviously, it isn’t. Luca broke ocg in the last hours of sunday, but it was instanly fixed. We just want to go bed, tomorrow is another day. WikiToLearn 0.7 is online, and we are ready to astonish you.

Daniele