Sunday, January 14, 2007

Reducing bottlenecks in the JEE stack - Network Overhead: Part 1.

Performance tuning in a typical JEE environment is an art. Like any other artform, it only gets better with experience. There are a lot of moving parts to consider - load balancers, JEE containers, application design - frameworks like ORM mappers and IOC engines, JVM performance - garbage collection configuration, database prformance, database access, disk IO. In a distributed environment, there are even more bottlenecks introduced - serialization overhead of marshalling and unmarshalling java objects, overhead of keeping the cluster coherent, overhead of avoiding the split-brain problem and so on. One of the most overlooked or least proiritized area is network overhead. Understanding how data is being moved across your servers and "how much" data is being moved across and then fine-tuning this network overhead can bring a tremendous performance boost to your application.

My motivation to write this article came from a recent visit I made to one of our customers. They have a distributed JEE middleware layer which uses protocols like JMS and RMI to move data across multiple JVMs running on multiple commodity hardware boxes. The JMS layer is used for sending asynchronous messages whereas the RMI layer is used for sending broadcast messages to multiple consumers. During very high usage, the network pipe gets extremely busy (up to 70%). This is a gigabit network we are talking about. Obviously, there was a concern of the network getting completely clogged at the rate at which the usage was growing. In fact, the network would get so busy at peak usage times, that they had to introduce artificial pauses and they couldn't afford to send real time messages any more. This is the reason I was there to demonstrate how Terracotta could reduce this overhead significantly. We dicided to replace the RMI-boradcast layer with Terracotta DSO as a pilot. Let's look at how RMI and Terracotta would work in this scenario to understand the test.

A normal RMI invocation works as follows: a client application invokes a method on an object on the server. The parameters are marshalled and sent out over the network to the server machine, where the computations are performed, results are marshalled and sent back to the client. The two main areas where delays occur are the marshalling (object serialization and deserialization) and the network overheads. Object serialization overhead is a lot higher if you are sending deep object graphs as parameters since a deep-clone serialization (and deserialization) needs to be performed. In the above mentioned case, each consumer JVM ran a RMI server and the client JVM (the broadcaster) would iterate through a list of consumer IPs or hostnames and send a message (invoke a method) on each of the consumer JVMs.

Terracotta is an open source clustering solution that works as follows: each clustered JVM is policed or managed by a central Terracotta server. Whenever a shared object is mutated (within the boundaries of a lock), the delta level fine grained (field level) changes are maintained, and propagated to other clustered JVMs if and when the same object is accessed. Since there is no serialization and deserialization involved, there is no marshalling and unmarshalling overhead - no matter how deep the objec graph is. Also, since Terracotta only ships fine-grained changes, the difference in the amount of data sent across the network can be huge depending on the case. All this is possible with Terracotta as it maintains physical object identity across the cluster (i.e. foo == bar is preserved, not just foo.equals(bar)).

Ok, all this is in theory. Let's look at the actual test now. To simulate the customer environment, I wrote a simple RMI application with 1 broadcaster node broadcasting to 3 consumer nodes an object graph which is 10 levels deep and is of total size 10k. For this result, the boradcast happened every 10 millisenconds and I used ntop to monitor network usage.

Results:


The RMI broadcast sent a total of 2.6 GigaBytes of data over one hour i.e. ~ 722 KiloBytes per second.

With Terrcotta, the overall network usage over one hour was 491 MegaBytes i.e. ~ 136 KiloBytes per second.

This is a 5.3 times improvement by using the Terracotta solution. Now if you scale these results to 40 consumers and 4 broadcasters (actual real world scenario) - the improvement is ~ 11 times.

Let's analyze these results. In this case, Terracotta is able to achieve such improvements because in the entire 10K object graph which is 10 levels deep, only one String value (100 bytes) is changing and being broadcasted. With the RMI solution, the entire 10K object graph is being broadcasted to every consumer node whereas Terracotta only sends the bytes that have changed.

If we consider a non-broadcast use case where the changes do not need to be sent to every other node in the cluster, Terracotta would do a lot better whereas a serialization based clustering solution would still send the entire object graph to every other node to keep the cluster coherent. (There are some serialization based clustering solutions that implement a buddy replication system or some other mechanism where the object graph is not shipped to every other node, but the drawbacks associated with those is outside the scope here.) Terracotta guarantees cluster coherency whithout having to send the changes to every peer in the cluster because the central server knows which object is being accessed on which node through the lock semantics and by preserving object identity. In fact, due to the object identity being preserved, Terracotta sends fine grained delta-level changes ONLY WHERE they need to be sent i.e. only to the JVM where the same object is accessed.

All of the above improvements are over and above not having to maintain any RMI code and working with natural POJOs only. I have not pointed out the performance gains in the above test seen as a result of no serialization and not having to marshall and unmarshall the 10K, 10 level deep object graph as that is outside the scope of this discussion. I wanted to point out the impact network overhead can have on a typical JEE application.

For those who are interested, in the next part of this article I will post the source code of the test and instructions on how to run it.

Links:
ntop
RMI
Terracotta

Download ntop.
Download Terracotta.

237 comments:

«Oldest   ‹Older   201 – 237 of 237
Anonymous said...

buy tramadol tramadol dosage - tramadol for dogs how much

Anonymous said...

buy tramadol online tramadol zoloft drug interaction - tramadol withdrawal one week

Anonymous said...

buy tramadol tramadol 50 mg para que es - order tramadol online forum

Anonymous said...

buy tramadol tramadol 300 er - tramadol online doctor

Anonymous said...

buy tramadol buy tramadol an 627 - tramadol ultram ultracet

Anonymous said...

tramadol online buy tramadol hong kong - tramadol pregnancy

Anonymous said...

carisoprodol 350 mg carisoprodol soma uses - carisoprodol class of drug

Anonymous said...

alprazolam without prescription xanax side effects for men - what is xanax good for

Anonymous said...

buy xanax online overdose symptoms for xanax - where to buy xanax online no prescription

Anonymous said...

buy xanax online xanax withdrawal after 3 days - xanax withdrawal emedicine

Anonymous said...

buy tramadol online tramadol 50 mg kapszula - buy tramadol no prescription online

Anonymous said...

buy tramadol online is it legal to order tramadol over the internet - tramadol withdrawal last

Anonymous said...

http://ranchodelastortugas.com/#93851 xanax bars online cheap - xanax withdrawal lightheaded

Anonymous said...

buy tramadol medication tramadol withdrawal forum - tramadol hcl pregnancy

Anonymous said...

http://staam.org/#92453 tramadol for sale no prescription usa - tramadol vs toradol

Anonymous said...

http://southcarolinaaccidentattorney.com/#54632 carisoprodol 350mg - carisoprodol side effects weight gain

Anonymous said...

buy tramadol online tramadol hydrochloride 100mg used - tramadol withdrawal exercise

Anonymous said...

buy carisoprodol no prescription soma carisoprodol schedule - carisoprodol 350 mg - 90 tabs

Anonymous said...

buy xanax online alprazolam 0.5mg side effects - xanax overdose what to do

Anonymous said...

http://bayshorechryslerjeep.com/#2902 xanax drug history - xanax 2mg side effects

Anonymous said...

cheap xanax xanax prescription online no prescription - xanax side effects drugs.com

Anonymous said...

is it illegal to buy xanax online order xanax 2mg online - many xanax pills can take

Anonymous said...



my web-site ... website

Anonymous said...

I have been exploring for a little bit for
any high quality articles or blog posts in this kind
of area . Exploring in Yahoo I ultimately stumbled
upon this website. Studying this information So i am happy
to show that I have a very excellent uncanny feeling I discovered
just what I needed. I most definitely will make certain to don?
t overlook this site and provides it a glance on a continuing basis.


My weblog teeth whitening

Anonymous said...

Hmm is anyone else experiencing problems with the images on this blog loading?

I'm trying to determine if its a problem on my end or if it's the blog.
Any responses would be greatly appreciated.


Also visit my website - garcinia

Anonymous said...

If you wish for to take a great deal from this article then you have to apply these methods
to your won website.

Also visit my site: raspberry ketone

Anonymous said...

With havin so much written content do you ever run into any problems of plagorism or copyright infringement?
My website has a lot of exclusive content I've either authored myself or outsourced but it looks like a lot of it is popping it up all over the web without my authorization. Do you know any methods to help prevent content from being stolen? I'd truly appreciate
it.

My website :: garcinia cambogia

Anonymous said...

Hello, this weekend is fastidious designed for me, for the reason that this time
i am reading this enormous educational paragraph here at my home.


My weblog whitening method

Anonymous said...

It's actually a cool and useful piece of info. I am glad that you shared this useful info with us. Please keep us up to date like this. Thanks for sharing.

Visit my web page :: best teeth

Anonymous said...

Attractive section of content. I just stumbled upon your site and in accession capital to assert that I acquire actually enjoyed account your blog posts.
Anyway I will be subscribing to your augment and even I achievement you access consistently rapidly.



Also visit my weblog: new zealand deer antler extract

Anonymous said...

With havin so much written content do you ever run into any issues of plagorism or copyright infringement?

My website has a lot of completely unique content I've either authored myself or outsourced but it seems a lot of it is popping it up all over the internet without my authorization. Do you know any solutions to help protect against content from being stolen? I'd certainly appreciate
it.

My weblog green coffee bean

Anonymous said...

I will right away grab your rss as I can not find your e-mail subscription hyperlink or e-newsletter service.

Do you have any? Kindly let me recognise so that I could subscribe.
Thanks.

Feel free to visit my blog post - teeth whitening

Anonymous said...

Because of financial restrictions many people might choose the less expensive insurance policy estimate since they're selecting what ever they can afford and because they cannot think that they will require everything extra cover. Did you know that you might end up spending more than the initial insurance coverage estimate? Here is the key reason why. dodatkowy odczyt 4. Contribution. In a situation exactly where two or more insurance providers is covering a particular danger, in case a reduction occurred, the particular insurance companies must contribute on the settlement of the state according to their own rateable percentage. kamagra jak zdobyc Standard home owners insurance does not protect the particular related costs of restoring water-damaged parts of your home. Hence, often keep your drinking water movement system is carefully preserved which leaks, actually small types, are usually dealt with quickly. Overflows or even unintended discharges out of your home's sump water pump plus sewer drain may wreck the particular flooring and other items in your own home. If you notice sluggish depleting or domestic plumbing problems, possess a professional examine and repair all of them before they worsen. kamagra jak zdobyc

Anonymous said...

Helps you get practical choices klik zasoby Where can you purchase PLPD Car Insurance? Because of technology as well as the increase from the internet shopping for a cheap PLPD insurance policy has never been less difficult. Evaluating rates plus shopping around was previously a pain. It all involved making many calls plus speaking with a half dozen product sales repetitions plus insurance companies. Obviously additionally you had to listen to each sales person put in their 2 pennies and make the sales hype. Fortunately, instances have got transformed and you understand longer need to do points the particular old-fashioned method. You can now compare quotes and even purchase PLPD auto insurance right from the convenience of your home. If it is your goal to get the cheapest cost feasible for PLPD insurance plan I recommend checking the web and going to an online insurance policy evaluation site. These sites will make your daily life simple. They are going to request a person a few simple queries the zip code as well as the kind of car a person generate. After you have offered this information the particular assessment web site will provide you with a list of the providers in your squat code using the lowest prices. When you have refined lower your selection you can finish your own quote using the service provider of your option plus finalize your own buy on-line or even having a live life consultant. levitra jakosc Your own homeowners insurance will cover your own deficits in the event of fire, hail, super, plus windstorm. But if an electrical spike classes through your neighborhood plus problems your own $4, 000 home theater program, you will need to hand on the cash to obtain it set or even changed. viagra pigulki

Anonymous said...

Cable Broadcasting Products For Telly Programming

my web-site :: Panasonic PT-AE8000

Anonymous said...



Feel free to visit my blog: web site

Anonymous said...

Acai powder functions a reasonable index that can help
in controlling excess weight. A bit of a beetle produces exceeding problems with Kona beverages facilities.

You might need to attempt out alternative smash facility to search for the one particular fulfils you.
Here we are at these content articles boasting involving tutoring throughout the offer you inside of a telephone finance
application. The frozen varieties will probably be growing
media cold gourmet coffee, frosty cappuccino and mocha.


Have a look at my website :: turkish coffee grinder machine

«Oldest ‹Older   201 – 237 of 237   Newer› Newest»