Django is great. No doubts about it. But - sometimes comes the question of scalability, especially in a time-and-budget constrains. Well - the very least we can say is that - our client suddenly decided that he needed about 200 rqs - yes, that’s 200 hundred requests per second - and yes, that’s 1728.000 requestes per day. Well, no problem in achieveing that - so far there’s no talk about budget. Well, there was, the all-dreaded least-cost-budget-criteria. So, ELCODO team stood up for the chalenge and came up with a few interesting, open-source ideas. First of them was the load balancing - here came the ideas about using vrrp/carp. The only real concern was about the instability of the IPv6 implementation - which was easily resolved through aggresive negotiations with the client :)
The second issue was of similar scale - a choice between using replication of the MySQL database and using a dedicated MySQL server. The dedicated option was chosen. So no architectural or otherwise programistic problems here. One database server and that’s enough for now.
The first of real problems came with the Django module consisting of some sort of an upload module - the uploaded files needed to be in an exact sync with every server.
For example - server1.elcodo.pl needed to have the same files ijn /upload/ dir as server2.elcodo.pl. The problem was really easy to resolve - given the use of a replicated, clustered filesystem such as GlusterFS. Under consideration, GlusterFS was the most robust, feature-full choice we could make in under a time limit of several minutes. So, a GlusterFS filesystem was created, using 4 servers as filesystem nodes.
The last problem is considered by many a no-problem situation, but for us is a really headache-case. The problem of “overall efect”, the situation in which you just spent 8 hourse building the systems and it’s simply too slow. The GlusterFS worked like a charm, giving us near real-time results. The database (not replicated) was working fine, as far as the MySQL settings allowed it). One would suggest that the only problem (besides implementing VRRP/Carp in a reasonable way) was the webserver himself. That is not true. We have considered an advanced Nginx (and eventually a Litghttpd) setup - but the Apache 2.2.* was enough. Using the mod_wsgi module without daemon process seems to be a preety acceptable solution.
So, to bring some “hard-evidence” and “true statistics”:
- you can easily serve ~200++ requests/sec with Apache/mod_wsgi on cheap hardware
- Networked, replicated filesystems - like GlusterFS - are really neat
- all you need is a good LoadBalancing. Good LB will get you about 70% problems less
- Have a fat pipe if you want to deliver “fat” content - no cache or otherwise cheating will help you.
Summing up - 4 servers + Django (mod_wsgi) + MySQL + GlusterFS + LoadBalancing.
3 hours of work. Hardware/instalation cost below 1000$. And that’s called OpenSource ;)
