Comments on: Big Iron-y http://www.ldscio.org/2007/03/21/big-iron-y/ Chief Information Officer for The Church of Jesus Christ of Latter-day Saints Mon, 26 Jul 2010 01:33:29 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Mr. Blue http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-990 Mr. Blue Thu, 03 May 2007 21:20:14 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-990 >>Finally they tried to upsell us to a higher quality SAN at a deep discount. You gotta love when they agree their own product is junk ;) Hope everything worked out. >>Finally they tried to upsell us to a higher quality SAN at a deep discount.

You gotta love when they agree their own product is junk ;) Hope everything worked out.

]]> By: Paul Hepworth http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-978 Paul Hepworth Mon, 30 Apr 2007 02:00:03 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-978 The only thing worse than not having redundancy is thinking you have it and then relaxing. Unless fail-over configurations are tested frequently they will not work when you need them. The only thing worse than not having redundancy is thinking you have it and then relaxing.

Unless fail-over configurations are tested frequently they will not work when you need them.

]]> By: Mr. Blue http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-967 Mr. Blue Thu, 26 Apr 2007 03:59:31 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-967 It certainly is worth it, if done right. The fact that it took 30 minutes to figure on the load balancer leaves me to question both the competence of staff as well as your monitoring capabilities or lack thereof. But i digress: * Written properly, monitoring scripts and scenarios can pinpoint easily the failure in your applications/systems. You need to monitor not only your balanced app, but also each individual app instance; doing so may have helped pinpoint failure at the balancer. * How is moving to a new vendor going to solve a poorly-architected system? Most SANs I've ever known of are out of the box crazily-redundant. Either your datacenter was hit by a nuke, or a monkey architected your SAN, plain and simple. <em>[Joel: Turns out the vendor agreed the SAN's were bad. We tried new drivers, new hardware components, etc, etc. Finally they tried to upsell us to a higher quality SAN at a deep discount.]</em> It certainly is worth it, if done right. The fact that it took 30 minutes to figure on the load balancer leaves me to question both the competence of staff as well as your monitoring capabilities or lack thereof. But i digress:

* Written properly, monitoring scripts and scenarios can pinpoint easily the failure in your applications/systems. You need to monitor not only your balanced app, but also each individual app instance; doing so may have helped pinpoint failure at the balancer.

* How is moving to a new vendor going to solve a poorly-architected system? Most SANs I’ve ever known of are out of the box crazily-redundant. Either your datacenter was hit by a nuke, or a monkey architected your SAN, plain and simple.

[Joel: Turns out the vendor agreed the SAN's were bad. We tried new drivers, new hardware components, etc, etc. Finally they tried to upsell us to a higher quality SAN at a deep discount.]

]]> By: Mark R. Bush http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-964 Mark R. Bush Fri, 20 Apr 2007 12:35:55 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-964 Testing! You can plan, prevent, and pretend to protect all day long! I've been working in HA environments for nearly 10 years now and if there is one lesson I've learned it's that you must test your failover/loadbalancing mechanisms. You typically test this behavior at the time of implementation. But what about 60, 120, 240 days into it? Regular, scheduled, failure testing is the only way I know to minimize (notice not eliminate completely) any risk in your HA environment. The second thing I've learned is people are the biggest problem. Your change management process needs to be disciplined, SA's need to trust and use it (one of the biggest challenges), and plans need to be made, tested, and followed. Most technical type IMHO are process oriented and if you can get them to trust the change process and follow it, it too goes a long way. Testing! You can plan, prevent, and pretend to protect all day long! I’ve been working in HA environments for nearly 10 years now and if there is one lesson I’ve learned it’s that you must test your failover/loadbalancing mechanisms. You typically test this behavior at the time of implementation. But what about 60, 120, 240 days into it? Regular, scheduled, failure testing is the only way I know to minimize (notice not eliminate completely) any risk in your HA environment. The second thing I’ve learned is people are the biggest problem. Your change management process needs to be disciplined, SA’s need to trust and use it (one of the biggest challenges), and plans need to be made, tested, and followed. Most technical type IMHO are process oriented and if you can get them to trust the change process and follow it, it too goes a long way.

]]> By: Jacob http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-952 Jacob Mon, 16 Apr 2007 06:16:47 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-952 I've found that in our data center it is always the thing we least expect to fail that does. That means that while redundant systems are good to prevent against foreseeable problems, they won't protect you against the problems you didn't anticipate. I recommend really good centralized monitoring software, so that you can determine very quickly what the less-anticipated problems really are, so you can fix them and get back online as fast as possible. You said: "It took us 30 minutes to realize our applications were down because of the failed load balancer." You could have reduced your downtime by bunch of minutes if you had scripts or monitors watching your load balancer. This sort of solution is generally much more affordable than the "figure out if the load balancer is less than robust and replace it with a more robust system if it is" situation. I’ve found that in our data center it is always the thing we least expect to fail that does. That means that while redundant systems are good to prevent against foreseeable problems, they won’t protect you against the problems you didn’t anticipate.

I recommend really good centralized monitoring software, so that you can determine very quickly what the less-anticipated problems really are, so you can fix them and get back online as fast as possible.

You said: “It took us 30 minutes to realize our applications were down because of the failed load balancer.” You could have reduced your downtime by bunch of minutes if you had scripts or monitors watching your load balancer. This sort of solution is generally much more affordable than the “figure out if the load balancer is less than robust and replace it with a more robust system if it is” situation.

]]> By: R. Alvez http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-900 R. Alvez Fri, 06 Apr 2007 01:56:23 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-900 I'm a Unix guy and as you probably know, Unix/Linux over abounds in tools. One tool I always run in my box is gkrellm, which incidentally has a plug-in to keep track of running servers. If they fail (as in a server becomes unavailable), I'll see it on the screen as well as I get a screen pop up or I can set it to send me an e-mail. So far it has never failed me, I'm always the first one to know if something "dies on action" and can take counter measures. Sometimes the most effective tools are the simplest of them all ;-) I’m a Unix guy and as you probably know, Unix/Linux over abounds in tools.
One tool I always run in my box is gkrellm, which incidentally has a plug-in to keep track of running servers. If they fail (as in a server becomes unavailable), I’ll see it on the screen as well as I get a screen pop up or I can set it to send me an e-mail.
So far it has never failed me, I’m always the first one to know if something “dies on action” and can take counter measures.
Sometimes the most effective tools are the simplest of them all ;-)

]]> By: Mario Hipol http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-757 Mario Hipol Tue, 27 Mar 2007 21:06:03 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-757 I recently had a personal experience that taught me something about redundancy. I had an issue a while back when installing a beta version of a "newly released office system" *ahem* When installing it asked me to reboot and I blue screened. I was extremely distraught. I had all our family photos, etc. I wans't sure what to do. It just so happened a month prior I was testing a product that basically does and ISO of your system and restores it exactly to that point. I searched all over and found that simple test file I had created on my system. It was a lifesaver. I have since come to realize that the importance of having things in place to recover information is extremely important even on a personal level. I recently had a personal experience that taught me something about redundancy. I had an issue a while back when installing a beta version of a “newly released office system” *ahem* When installing it asked me to reboot and I blue screened. I was extremely distraught. I had all our family photos, etc. I wans’t sure what to do. It just so happened a month prior I was testing a product that basically does and ISO of your system and restores it exactly to that point. I searched all over and found that simple test file I had created on my system. It was a lifesaver.

I have since come to realize that the importance of having things in place to recover information is extremely important even on a personal level.

]]> By: WadeA http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-742 WadeA Mon, 26 Mar 2007 19:35:53 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-742 People. Process. Technology. We've all heard it before - I think - yet we often fail to put systems and methodologies in place that actaully monitor all three. Sometimes it isn't about adding more protection and complexity, it's about simplifying and watching what exists more closely or more effectively to see when there are symptoms and recover more quickly. You also have to be willing to invest in all three. Skimping on one usually means you'll be reinvesting back into one of the other areas - skimp a few dollars of the technology and you'll likely invest in people and process to keep it in play. Neglect the growth of your staff and there's a good chance you'll be adding process or technology attempting to compensate. Keep in mind that symptoms don't always point you to the area that is the REAL cause. So what do you do to break the cycle? Take steps to understand where you are today. Are your existing process effective in supporting your business objectives. Are your people able to use them effectively in their roles. Examine the source of your outages - I mean the REAL source. Just like any self-critique, you're going leanr stuff you don't like and aren't fun to admit, but it's the only way to really get on the path to improvement. I don't want to ramble on here so lastly I'll just say, don't be afraid to get some outside help. Being the "self-reliant" culture that we are it's not something that we're very good at doing or knowing when to do it. That "objective" perspective can often provide valuable insight that can be learned in no other way. People. Process. Technology.

We’ve all heard it before – I think – yet we often fail to put systems and methodologies in place that actaully monitor all three. Sometimes it isn’t about adding more protection and complexity, it’s about simplifying and watching what exists more closely or more effectively to see when there are symptoms and recover more quickly.

You also have to be willing to invest in all three. Skimping on one usually means you’ll be reinvesting back into one of the other areas – skimp a few dollars of the technology and you’ll likely invest in people and process to keep it in play. Neglect the growth of your staff and there’s a good chance you’ll be adding process or technology attempting to compensate. Keep in mind that symptoms don’t always point you to the area that is the REAL cause.

So what do you do to break the cycle? Take steps to understand where you are today. Are your existing process effective in supporting your business objectives. Are your people able to use them effectively in their roles. Examine the source of your outages – I mean the REAL source. Just like any self-critique, you’re going leanr stuff you don’t like and aren’t fun to admit, but it’s the only way to really get on the path to improvement.

I don’t want to ramble on here so lastly I’ll just say, don’t be afraid to get some outside help. Being the “self-reliant” culture that we are it’s not something that we’re very good at doing or knowing when to do it. That “objective” perspective can often provide valuable insight that can be learned in no other way.

]]> By: Bill Coulam http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-721 Bill Coulam Sun, 25 Mar 2007 05:11:26 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-721 The goal is certainly noble, and should be do-able. But in practice, the sales pitches almost never measure up. We once had some Sun clustering solution for our 12 CPU twin E4500's, and the only serious problems we ever really had with those machines were due to the clustering. We gave it more than it's share of chances to measure up. Once we finally gave up and removed the cluster, we finally got to enjoy some of that legendary Sun reliability and never experienced unexpected downtime again. Our next Tech Director purchased a Hitachi SAN that seemed to work alright. Experiences with other HA solutions similar to yours seem to be the status quo, including horror stories around Raid 5 and tons of wasted effort and money on Oracle RAC. There is a semi-formal "club" of sorts, composed of many of the biggest names in Oracle, dedicated to stamping out all incarnations of Raid 5 (http://www.miracleas.com/BAARF/BAARF_members_sql.php). So testing the brains out of one's HA solution seems to be the best recourse. The goal is certainly noble, and should be do-able.

But in practice, the sales pitches almost never measure up.

We once had some Sun clustering solution for our 12 CPU twin E4500’s, and the only serious problems we ever really had with those machines were due to the clustering. We gave it more than it’s share of chances to measure up. Once we finally gave up and removed the cluster, we finally got to enjoy some of that legendary Sun reliability and never experienced unexpected downtime again.

Our next Tech Director purchased a Hitachi SAN that seemed to work alright.

Experiences with other HA solutions similar to yours seem to be the status quo, including horror stories around Raid 5 and tons of wasted effort and money on Oracle RAC. There is a semi-formal “club” of sorts, composed of many of the biggest names in Oracle, dedicated to stamping out all incarnations of Raid 5 (http://www.miracleas.com/BAARF/BAARF_members_sql.php).

So testing the brains out of one’s HA solution seems to be the best recourse.

]]> By: Dan Fitzpatrick http://www.ldscio.org/2007/03/21/big-iron-y/comment-page-1/#comment-717 Dan Fitzpatrick Sat, 24 Mar 2007 19:18:11 +0000 http://www.ldscio.org/2007/03/21/big-iron-y/#comment-717 In my experience computer hardware is like race car hardware - the harder you push it the faster it breaks (specifically disks). My solution to reliability to to over-provision hardware so it rarely exceeds 50% utilization. Kind of like an F1 racing at Nascar. If disks are reading and writing at capacity 24/7, they will wear out much faster than if they run at less than 50% capacity. For disk reliability and scalability I prefer RAID 10 with disks from different manufacturers on each RAID 0 set. I agree with Phillip Cox that fail-over testing is essential. It is our policy that we use our fail-over systems during all system upgrades. We fail-over, do the upgrade of the primary system, and return the primary system back to production, then upgrade the fail-over system. You kill two birds with one stone by testing fail-over and upgrading in the same process. This is basically training the staff to deal with system failures as a normal upgrade (which would have to occur anyway to fix whatever broke.) In my experience computer hardware is like race car hardware – the harder you push it the faster it breaks (specifically disks). My solution to reliability to to over-provision hardware so it rarely exceeds 50% utilization. Kind of like an F1 racing at Nascar. If disks are reading and writing at capacity 24/7, they will wear out much faster than if they run at less than 50% capacity. For disk reliability and scalability I prefer RAID 10 with disks from different manufacturers on each RAID 0 set.

I agree with Phillip Cox that fail-over testing is essential. It is our policy that we use our fail-over systems during all system upgrades. We fail-over, do the upgrade of the primary system, and return the primary system back to production, then upgrade the fail-over system. You kill two birds with one stone by testing fail-over and upgrading in the same process. This is basically training the staff to deal with system failures as a normal upgrade (which would have to occur anyway to fix whatever broke.)

]]>