At ARNES, we have been following news about the spread of the novel coronavirus since early March. Initially, we were spreading coronavirus-themed humor over morning coffee in our offices, only to fall silent just a few days later. Somewhat apprehensively, we instead started browsing news portals before every planned business trip abroad. Looking back, we can see that events escalated rather quickly – in just over two weeks, we were immersed in a new reality, using gallons of disinfectant, issuing warnings about hand hygiene, and, literally overnight, moved our entire team to home office.
As the situation worsened, we increasingly started thinking about what to do in the event that schools and universities closed and classes were moved entirely online. This is a big challenge and a scenario that is hard to prepare for. Where will the bottlenecks be, how will we be able to eliminate congestion, and what will working from home for a large part of the country’s population mean for the stability of the country’s internet? After the lockdown, remote work and distance learning, we recapped the events.
We noted an increased use of services already before Monday, March 16th, when distance learning was introduced as many started with preparations during the weekend. A tidal wave of interest that hit our remote learning services, Arnes Učilnice (Moodle), Arnes Splet (WordPress), Arnes Video portal, video conferencing services and Arnes SPM did not come as a surprise. On Monday, we saw an up to 100-fold increase in the number of users of ARNES cloud services.
Before the pandemic, Arnes Učilnice ran on two servers that easily supported 150 concurrent users. The short time frame did not allow us to plan properly; by Friday we realized that increasing the resources of existing virtual servers on Monday would by no means suffice. In three days, we therefore turned the entire server architecture upside down, moving from two virtual servers to 7 physical and three virtual ones. We just learned as we went, really, about how Moodle works in such a layout. The service administrator and those who normally performed other tasks took on the additional burden without hesitation and coordinated by videoconferencing. On Sunday, we thought that we had made it.
On Monday, the number of users kept increasing at an incredible rate, like a real tsunami. The service architecture should be correct, there were increasing numbers of users, and despite the fact that we added new servers, they were overwhelmed. And they should not have been.
In such unprecedented circumstances, it is difficult to get a large enough number of servers. We had some that were not yet in use at hand at ARNES, and we moved additional ones from other services. We would like to use this opportunity to extend our sincere thanks to Jožef Stefan Institute, the companies Žabec, Xenya and DHH, as well as the Slovenian postal services (Pošta Slovenije), who generously offered their own equipment and experts entirely free of charge. They provided them within less than 24 hours of our initial contact! Servers that were borrowed for Arnes Učilnice from Jožef Stefan Institute are completely new and intended for high-performance computing – for supercomputers. The graphics cards of the Jožef Stefan Institute support the operation of Arnes Video portal.
From Friday to Wednesday, ARNES employees worked for 15 hours straight until they could not any longer. Engineers worked in pairs, two were connected to the server. One member of the pair performed work and explained the procedure to his co-worker, who assisted and conducted supervision to ensure that the process was error-free. Such a complex service is difficult to manage without something slipping your mind under pressure. We reviewed every detail, measured the impact of every change. The performance of Arnes Učilnice was continuously improving, but it still was not flawless. To facilitate work, communication took place by videoconference on ARNES’ Pexip system.
On Wednesday night, we discovered a critical bug in the Arnes Učilnice code and the response time for users dropped to less than half a second at that time. The programmer will receive an extra-large pizza for all his efforts! However, the work was far from completed, we were confronted by a new challenge: the infrastructure for the AAI was not able to process a large number of logins in such a short time. Troubleshooting minor issues for the various services, which the users actually may not have even noticed, took a few more days. We also focused on user experience, as we had to turn off some service functionality due to resource overloads.
At the same time, we detected and solved problems with the operation of other services. As fate would have it, Arnes mail service experienced one of the biggest DDOS attacks thus far, which lasted for 12 hours straight, continuously burdening the server with 300 emails per second.
The attack was much more sophisticated than what we normally see. We were unable to restrict it to specific domains, users, or IP addresses. The addresses from which the attack was launched originated from 2,700 different subnets, so we temporarily restricted traffic from those countries which typically do not harbor connections to our servers. In this way, we reduced the power of the attack to 50 messages per second, which allowed us to maintain our responsiveness for emails unrelated to the attack.
Distance learning still takes place according to a regular schedule, mostly in the morning, with users accessing the services at the same time. Naturally, this takes a heavy toll on the server infrastructure and connections to users. By using the Arnes Video educational portal, which allows teachers to upload recordings of their lessons and publish relevant links in online classrooms and on school websites, we managed to lessen the burden and distribute it evenly.
There has been an almost 30-fold increase in daily number of video viewings. More videos have been uploaded in only three months of this year than in the entire 2019. On Saturday, March 21st, however, an alarm went off in the evening. The Arnes Video portal was disabled by 7,000 viewers who wanted to watch the film All Against All free of charge in the Slovenian Film Database. This political thriller, starring Peter Musevski, is set in a fictional town of Rovte under idyllic Alps and talks about democracy, electoral fraud and social disintegration. It goes without saying that we immediately increased our capacities and, in agreement with Filmoteka, made it possible to watch the film on Sunday as well.
The number of multimedia service users has been growing rapidly. Arnes VOX web conferencing is often fully occupied by the users already in the morning, so we redirect them to live streams. Lectures can be attended by over a thousand users at the same time. What is more, we offered free use of Cisco Webex video conferencing services, and we also prepaired Jitsi video conferencing for our users, which operate in Arnes Učilnice.
In Italy, the hardest hit country in Europe, articles transpired during the epidemic claiming that Italian operators were on the brink of a collapse as users quarantined at home overloaded their networks. In Slovenia, too, providers increased the amount of data available to users, unlocked software packages, while the users began to communicate by videoconferences, use video portals, educate themselves online and play games. How did the Slovenian internet react to this?
As early as on Friday, March 13th, we noticed a rapid increase in traffic through the SIX, which we manage at ARNES and which is intended for local traffic exchange among Internet providers in Slovenia. On Saturday, traffic exceeded 100 Gbit/s of total traffic for the first time in history, reaching 130 Gbit/s on March 22. In a bit over a week, the amount of traffic doubled.
On Saturday and Sunday, we were reaching out to all providers whose traffic reached connection limits in every possible way. Their response was swift, so most connections were upgraded as early as Monday, and the last one on Tuesday. In just a few days, joining forces, we increased the connection capacity by 110 Gb/s, and upgraded both ARNES network connections in SIX from 10 Gbit/s to 20 Gbit/s.
The Slovenian Internet is generally in good condition, and there is no fear of it ceasing to operate; naturally, however, there were occasional infrastructure congestion on some connections from providers to households. We find that most traffic occurs between seven and ten o’clock in the evening, so we recommend that you perform important tasks at other times.
The work we have done would not have been possible without laying foundation for working from home for around 100 employees. Most of our employees have business laptops, and for technical support we secured additional computers for work from home in the days before the lockdown. However, we encountered problems with the work of our call center consultants, as we originally thought that all work would be carried out at ARNES. When all public transport was eventually stopped, we had to find a way to take calls working from home.
We managed to arrange this only on Monday, which is why the call center had solely two very busy phone operators available on March 16th. Unfortunately, on that day, we were therefore unable to respond to a large number of users calls. The number of user calls increased five-fold on the first day of remote learning, and later stabilized at twice the average. We saw a six-fold increase in the number of email inquiries on Monday, which later plateaued at three times the pre-lockdown average. In the period between March 16th and March 25th alone, we received 1,346 reports of problems by email, which is more than in February, when we counted 1,280.
Domain names are one of the basic building blocks of the Internet. With the Slovenian .si domain, we also ensure the operation of such important ones as Ljubljana Medical Faculty’s mf.uni-lj.si, Ljubljana University Medical Center’s kclj.si, Maribor University Medical Center’s ukc-mb.si, and that of the National Institute of Public Health, nijz.si. The employees of Register.si, which is part of ARNES, take care of domain registrations and manage the distributed DNS system at home and abroad, which enables the accessibility of the entire Slovenian domain space. The operation of your websites, e-mail addresses and other services under .si depends on this, and it became ever more so important in these unprecedented times. Therefore, we call on all domain holders to check the validity of their domains via the WhoIs search engine and renew them in a timely manner through the registrar. This will ensure a smooth online communication of companies.
Expired .si domains are quarantined. You heard it right, domains, not just people, need to be quarantined, too. This quarantine is not related to the virus and has been in use for 15 years. After the quarantine expires, the domains are deleted from the system and again become available for registration. As of March 23rd, this quarantine period was extended from 30 to 60 days. Domain holders will thus have more time to renew their domains and at the same time greater security that their expired domain will not be taken by someone else.
As always, there are “entrepreneurs” who try to take advantage of a state of emergency for personal gain. In cooperation with SI-CERT, we therefore monitor domain registrations that contain words related to COVID-19, as under some other top-level domains, they are already being exploited to distribute malicious code or mislead web users. We also exchange data within the Council of European National Top-Level Domain Registries (CENTR).
Moving the office to our homes is associated with various challenges, including security. How do we access data from our office network, are our devices and home network secure enough? In these changed circumstances, what happens when our kid steps on our tablet or when our cat takes a stroll across the keyboard while we are in the kitchen having a cup of tea, and a business spreadsheet with important business data was inadvertently left open?
SI-CERT, the National Cyber Security Response Center, which is part of ARNES, has published guidelines for employers, companies and organizations, as well as for employees who work remotely in the current situation. They were well-received. SI-CERT employees work around the clock in this situation as well, of which we have informed all providers of essential services and the Information Security Administration of the Republic of Slovenia, while also adding instructions for reporting incidents. At SI-CERT, we have added a list of risks and the most common scenarios of attacks and abuse. Of course, with special focus on medical facilities, to whom we shall always offer help as soon as it is requested. We keep our fingers crossed that we won’t have to intervene!
We invite you to follow the new contributions on the varninainternetu.si portal and social network channels. We are preparing new video instructions and practical tips on how to identify online risks! If you sign up for a newsletter there, you will receive it in your mailbox.
The pressure faced by ARNES employees was tremendous. We had to provide distance learning to 300,000 schoolchildren. The work was demanding as only the most essential staff could be at ARNES. Nevertheless, it was carried out continuously and the employees were keeping their cool. At all times, ARNES employees were ready to offer a helping hand to those who needed it, sharing their invaluable knowledge and offering information and logistics support in an appropriate and calm manner. An important incentive was the moral support received from our users.
We are still upgrading our services, and we did not have time to celebrate, except for sharing some joyful moments via a video conference and in Arnes Utrip, a newsletter that highlights key achievements of our colleagues in isolation.