Skip to content

Assume the Worst — and Verify Your Success

There are very few “rants” that I go off on when it comes to programming because I generally believe that most programmers want to do the right thing and there are innumerable ways to approach any problem, so who am I to say better?

That said, there are a few hot-button issues for me when it comes to programming patterns, and I encountered one such situation again recently. To frame the topic, here’s what I encountered:

Some machines within our domains at work are centrally managed using Microsoft System Center products (more specifically here, System Center Configuration Manager) and overall the system works pretty well. The basic premise is to ensure (enforce, really) a certain level of software version/patch compliance and this is meant to be achieved by having SCCM push down a package which verifies local software and automatically launching patches and product installations. The problem comes when something goes wrong with these installations and because of poor (perhaps “junior developer”) implementation patterns, this process which is meant to make things better can be detrimental instead, to the point of even acting like a denial of service attack on our own managed resources!

So what are the problematic patterns here I stamp my fist on tables and rant about?

  • Assuming success is the natural outcome
  • Assuming any arbitrary failure can be recovered from

In this particular case, the patching solution was making things worse because it attempted to install a piece of software which was inappropriate for the environment, failing, and then retrying ad-infinitum. This infinite loop not only meant that the machines were not patched as intended (and actually got further and further behind in patching because the process never completed successfully), but also ate up most of the machines processor and I/O constantly trying to install a package which would never succeed!

This situation was particularly annoying because it clearly should have been caught in QC before ever impacting my machines and demonstrates some lazy error handling practices which assumed success and swallowed errors rather than recovering or bailing out.

The definition of insanity is doing the same thing over and over again and expecting different results. - Albert Einstein

Blindly Assuming Success

This is a pattern I see quite frequently in software. I pride myself on being an optimist, but there is optimism and then there is an unhealthy disconnection from reality. Mechanical and structural engineers working in the “real world” have to base their designs on known, proven tolerances, planning for failure and understanding the limits of their materials. They know that given the environment their design is supposed to operate in, they can account for the expected stresses plus some additional safety factor. Outside of this tolerance range is unpredictability, “out of warranty”: risk. I often see the exact opposite in software design: there is a blind faith that whatever is attempted will succeed. So I rant:

Assume Failure — and Verify Your Success!

Does this mean you must verify the outcome of every statement? Of course not — we must always balance the risk of failures against their likelihood of occurring, and incredibly basic operations have a very high likelihood of success but I’m always amazed at the assumption that resources are infinite and immediately available. “On my dev box I was always able to create file X, so why shouldn’t I be able to do so in production?” cries the Dev. You might be right, but let’s be certain, shall we? Just verify the operation you attempted after doing so — or better yet, verify that it should succeed before even attempting it. What is the worst that can happen by assuming failure? Typically, just a disappointing experience for the user because what they wanted to occur did not. On the other hand, by assuming success without verifying can (and has) lead to some catastrophic situations where not only does the program not do what the user desired, but much worse damage occurs as well.

Failure is Always an Option

Contrary to the famous quote of Gene Kranz during the Apollo 13 mission, in software failure is always an option. Your own code will never know enough about its own environment and context to be truly bulletproof — nor does it need to be. What it does need to explicitly accept, though, is that failures can happen: some we can expect, understand, and may even be able to gracefully recover from, and all others we cannot. It’s the second class of failures that seem to be downplayed — not that we don’t recognize they can happen, but rather that we should not even try to deal with them. Ever seen code like this?:

try {
   // Do something
}
catch (Exception ex) {
   // Recover from ex
}

This kind of thing drives me nuts: it attempts to recover, but without any chance of really doing so, because catching such a generic exception type leaves you open to ANYTHING going wrong — and I guarantee you cannot recover from every failure type 🙂 This is the most extreme case, but even in many cases where more specific exceptions are caught there is still not enough context to reliably recover automatically. So if you can’t really recover, why are you even catching an exception?! So final rant:

Catch only what you expect can fail and that you can recover from — or fail gracefully and quit.

OK — ’nuff said, I’ll get off my soapbox. Over-and-out.

Advertisements

Taste Experiment: Grilled Chicken Broccoli Fettuccine Alfredo

Ingredients

  • 8 oz. egg fettuccine
  • 1 cup heavy cream
  • 1 stick unsalted butter, cut into pieces
  • 3/4 cup Parmigiano-Reggiano, finely grated
  • 1 Chicken Breast
  • Broccoli, 1 large head
  • 1/4 teaspoon salt
  • 1 teaspoon pepper

Preparation

Grille chicken until done. Set aside to cool. Cut into 3/4″-1″ thick slices.

Cut broccoli into florets. Steam for 10 minutes, then set aside.

Cook fettuccine until al dente. Reserve 3/4 cup cooking water, then drain.

Meanwhile, bring cream and butter to a simmer in a large heavy skillet over medium-low heat, adding 1/4 teaspoon salt and 1 teaspoon pepper. Add fettuccine, 1/2 cup reserved water, and cheese to sauce and toss. Add chicken and broccoli, mixing until coated.

Serve.

Result

Taste Experiment: Chicken Strips with Jalapeño Ranch Dressing

Ingredients

Chicken

  • 1 Chicken Breast, sliced into 1/4″ think strips

Brine

  • 8 cups water
  • 1/4 cup salt

Breading

  • 1 cup all-purpose flour
  • 1 1/4 cups plain bread crumbs
  • 2 tsp salt
  • 1 tsp granulated sugar
  • 2 tsp finely ground black pepper
  • 1/2 tsp ground white pepper
  • 1/2 tsp paprika
  • 1/2 tsp ground savory
  • 1/2 tsp ground sage
  • 1/4 tsp ground ginger
  • 1/4 tsp ground marjoram
  • 1/4 tsp onion powder
  • 1/4 tsp garlic powder
  • 1/4 tsp ground cayenne pepper

Dredge

  • 2 eggs
  • 1 cup milk
  • Canola Oil Spray

Dipping Sauce

  • 16 oz Fresh Ranch Dressing
  • 1 4oz can diced jalapeño chiles

Preparation

  1. Dissolve 1/4 cup salt in 8 cups water. Add chicken and marinate for 2 hours. Remove chicken from brine, rinse with water, and blot dry. Preheat oven to 375° F.
  2. Make the breading by combining all breading ingredients in a large bowl.
  3. In a separate bowl, beat the eggs and then stir in the milk. When the oven is hot, dip each piece of chicken in the egg and milk mixture and then press into the breading. Toss each chicken piece in the breading until well-coated. Let chicken sit for 2 minutes in the breading. Shake off the excess breading and arrange on a baking sheet that has been thoroughly sprayed with vegetable oil non-stick spray.
  4. When all of the chicken has been breaded, spray each piece with the oil spray until the breading is completely moistened.
  5. Bake chicken strips for 15 to 20 minutes or until nicely browned.
  6. While the chicken is baking, chop diced jalapeños fine. Combine Ranch dressing and chiles, mixing thoroughly.

Result

 

(Last) CenturyLink

When I moved into my house a few years ago, I had the typical set of move-in chores to complete: set up accounts for utilities, subscribe for pay-TV, find an Internet provider. In my area (a major metropolitan neighborhood), the geographic monopoly of internet access is controlled by Comcast (for cable connectivity) and Qwest (for old analog and DSL connectivity). At that time I was already aggrevated by Comcast’s customer "service" (aka "bend over, take it, and be happy we gave you anything") because they had flat-out stood me up for multiple past scheduled service calls, but I liked their network speeds and reliability so I called and set up an appointment for the "hook up". Yet again, Comcast no-showed and then acted like it was my fault for wasting my day waiting on them and tried to schedule another "sometime between 8 and 5" waste of time. I was fuming. Ok — no Comcast; they can go ****** a ***** before I spend another 8 hours waiting. Instead, I switched to the only other option: Qwest & DirecTV.

Operating at Qwest-Speed
A Shaky Foundation

Phone service enabled, DSL modem received and set up, and I thought I was connected. Oh how naive. 1st noticeable difference: the speed. OK — not a big surprise. I know DSL is an outdated and dead technology hindered by aging infrastructure. As long as I can get ~1.5 Mbps (below the lowest end Qwest offered), it’s good enough for browsing, mail, remoting into work, gaming. Unfortunately, although sold as 2Mbps (and paid for), my service rarely (if ever) even approached that. Frequently, speeds were sub .5 Mbps (yes — there is a decimal point there). In addition, I learned that my connection would frequently (like 10 to hundreds of times in a day) simply drop for several seconds to several hours. For you math geeks out there, Yes, that’s 0 Mbps. I called Qwest to get it resolved and they scheduled another day to be wasted waiting.

Legacy of Disservice

The agreed upon Saturday arrived and the technician (a nice enough fellow) appeared slightly before the allotted 4 hour appointment window expired. He spent 1 minute checking the (public) lines coming to my house and found that the signal at that point was already so degraded as to be basically useless. He then informed me that since my neighbor’s car was parked beside the telephone pole he needed to climb to check the signal further "upstream", he could not do any more work. I agreed that I’d ask the neighbor to move the car when he returned home, and the technician said he’d return the next day to continue.

Later that day, I had my neighbor move his car and parked mine to block the spot and ensure it could be easily freed when the tech returned. The next day (Sunday) was wasted spent waiting for the tech to return, and by 5:00 PM it was becoming clear that I was either forgotten or completely ignored yet again. I called Qwest and was informed that the ticket for my service had already been closed. I asked for an explanation as to 1) why the ticket was closed and 2) why had no one contacted me to let me know not to wait all day for a tech who had no intention of returning.

The "explanation" floored me: the ticket had been closed because someone had decided that the (again public) phone lines coming to my house/area could not handle the 5 Mbps service I was paying for, so they simply reduced my service speed and considered the matter closed. No followup communication. No checking to see if this resolution was acceptable to the customer. No offer of lowered rates or refund of payment for past months of terrible service. This is like an auto mechanic deciding that to fix a flat tire he should just remove the wheel entirely — and not tell the driver.

At this point I was stuck between an incompetent (Qwest) and an arrogant jerk (Comcast), with the only other option being no Internet connectivity, which today is not an option in my business. With the reduced traffic speed, I at least had connectivity again — bad, unreliable connectivity which was roughly 1/5 the speed I had received over a decade earlier, but connectivity.

This lasted all of about 1 month before I was again losing my connection consistently and constantly. It was to the point now that I wrote software specifically to monitor my connection and log all major failures to cite each time I had to call support. Luckily for Qwest, I can’t afford many 4-8 hour blocks of time waiting to see if a technician will show up, so I didn’t call for service often — I struggled through with the poor connectivity I had.

Sample connectivity log:
06:43 PM Tue 04/17/2012 : Connection lost.
06:56 PM Tue 04/17/2012 : Connection established.
06:58 PM Tue 04/17/2012 : Connection lost.
07:02 PM Tue 04/17/2012 : Connection established.
07:04 PM Tue 04/17/2012 : Connection lost.
07:07 PM Tue 04/17/2012 : Connection established.
07:09 PM Tue 04/17/2012 : Connection lost.
07:16 PM Tue 04/17/2012 : Connection established.
07:19 PM Tue 04/17/2012 : Connection lost.
07:19 PM Tue 04/17/2012 : Connection established.
07:24 PM Tue 04/17/2012 : Connection lost.
07:25 PM Tue 04/17/2012 : Connection established.
07:28 PM Tue 04/17/2012 : Connection lost.
07:29 PM Tue 04/17/2012 : Connection established.
07:34 PM Tue 04/17/2012 : Connection lost.
07:42 PM Tue 04/17/2012 : Connection established.
07:50 PM Tue 04/17/2012 : Connection lost.
07:51 PM Tue 04/17/2012 : Connection established.
07:54 PM Tue 04/17/2012 : Connection lost.
08:00 PM Tue 04/17/2012 : Connection established.
08:25 PM Tue 04/17/2012 : Connection lost.
08:28 PM Tue 04/17/2012 : Connection established.
08:30 PM Tue 04/17/2012 : Connection lost.
08:31 PM Tue 04/17/2012 : Connection established.
08:43 PM Tue 04/17/2012 : Connection lost.
08:47 PM Tue 04/17/2012 : Connection established.
04:33 AM Wed 04/18/2012 : Connection lost.
04:40 AM Wed 04/18/2012 : Connection established.
04:40 AM Wed 04/18/2012 : Connection lost.
04:43 AM Wed 04/18/2012 : Connection established.
10:08 PM Wed 04/18/2012 : Connection lost.
10:09 PM Wed 04/18/2012 : Connection established.
10:25 PM Wed 04/18/2012 : Connection lost.
10:25 PM Wed 04/18/2012 : Connection established.
06:13 PM Thu 04/19/2012 : Connection lost.
06:15 PM Thu 04/19/2012 : Connection established.
06:38 PM Thu 04/19/2012 : Connection lost.
06:45 PM Thu 04/19/2012 : Connection established.
07:04 PM Thu 04/19/2012 : Connection lost.
07:04 PM Thu 04/19/2012 : Connection established.
08:35 PM Thu 04/19/2012 : Connection lost.
08:41 PM Thu 04/19/2012 : Connection established.
More of the Same: aka LAST CenturyLink

Again I encountered terrible service to such a degree that even if the modem could acquire a connection, it was unusable. This latest round of failure’s cause was apparently that Qwest had been acquired by CenturyLink (to much advertising fanfare) and although the same employees carried over through the merger, their memory did not: someone in the telecom thought it a good idea to "upgrade" me back to a higher speed which the lines still could not handle. I was "upgraded" from sub-1 Mbps crappy connectivity to no Internet at all. Yet again I was on the phone with support (this time at least I had large logs of the numerous connection failures documented), and yet again the "solution" was to give me less bandwidth (still with no apology or reduction in cost, etc.). It seemed pretty clear that the company should simply fix the infrastructure or admit that they are incapable of delivering service and stop offering (and taking people’s money), but what do I know!? I guess I just have a conscience when it comes to screwing over customers…

So that’s it; I’m done with Last CenturyLink. They apparently can’t deliver the services they are selling and are continuing to push a dead technology. I believe in the power of competetive business to drive great value for customers — the best products and services should win, and those that don’t do right by the customer should fail and go out of business. The only reason companies like Qwest/CenturyLink are able to survive is because of the necessity of the services they control coupled with their existence under geographic monopolies where there is insufficient customer choice and competition.

So: I’m back to the arrogant Comcast. At least they only have to turn on the service and I can install the modem myself. Maybe I’ll schedule a 4 hour block during a workday next week and then not show up if I want to feel appropriately "serviced".

Taste Experiment: Kickin’ Split Pea and Sausage Soup

Tonight’s experiment: to liven up my standard split pea soup. Note that this is experimental, so I’ll come back and update ingredients, prep, etc. as necessary depending upon the results.

Ingredients

  • Dried Peas, 1 lb
  • Link sausage, 8 oz (e.g. summer or smoked sausage)
  • Carrots, 2 large
  • Onion, 1/4 large or 1 small
  • 2 large Anaheim peppers
  • Mexican Oregano Leaves, 1 Tbsp
  • Summer Savory, 1/2 Tbsp
  • Sage, 1/2 tsp
  • 1 qt chicken stock
  • 1 qt water
  • Salt, to taste

Preparation

  1. Rinse peas, letting rest in water
  2. Chop sausage into pieces (1/2″ thick discs, then these into quarters)
  3. Peel carrots and chop into 1/2″ thick discs, and dice onion and pepper, maintaining seeds.
  4. Sauté sausage pieces until brown
  5. Drain peas, and while cooking sausage, bring chicken stock, water, peas, carrots, onion, peppers, and spices to an open rolling boil.
  6. Add sausage and continue boiling, covered, over medium-high heat for 30 minutes, stirring occasionally.
  7. More (hot) water can be added during boiling to keep contents covered
  8. Remove from heat, allow 10 minutes to cool, and serve. Can also be topped with sour cream or créme fraiche

Result

Dinner is served!

A couple of insights for the next time around:

  1. Include salt initially (~1/2 tsp); was added after serving this time and could use it
  2. Although tasty and hearty, could use more kick. Next time around, might use 1 Anaheim and 1 Jalapeno, or perhaps add ground cayenne pepper

Taste Experiment: Stuffed Pork Tenderloin

Ingredients

  • 1 Pork Tenderloin 3/4 – 1 lbs
  • Blue cheese crumbles, 1/4 lb
  • Garlic, ~7-10 medium cloves
  • Black pepper, to taste

Preparation

Peel garlic. Unless they are very large, leave cloves whole.

Cut the pork tenderloin open, lengthwise. (Cut as if splitting the meat lengthwise, but only cutting 1/2 – 2/3 of the way through)

Season with pepper, to taste. Place garlic cloves in tenderloin and cover with blue cheese crumbles, filling the meat; using wet fingers can help stop the cheese sticking to them as the meat is stuffed.

Truss with cooking twine, if necessary.

Place in preheated oven for 35 minutes at 350 degrees.

Remove from oven and leave to rest for ~ 3 minutes

Remove twine (if necessary). Slice into medalions, roughly 1″ thick.

Result

 

Baseboards: Cutting is Done!

Baseboards: all cutting and fitting is finally done — yay!!

With just a little bit left, it looks like my measurements for material were spot on.

Now I just have lots of cleaning, tacking, painting, and furnishing to do. It never ends…