UnfortunateShit.com

I had a shitty day at work, and worked right up until dinner was ready. While eating dinner, we noticed that the table light wasn’t acting right.  It turned on and off seemingly randomly, and then started strobing a bit.  The on/off action was way faster than would be possible if it was from remote commands/automations.  Most smart switches have an “air gap” tab just underneath the switch, which can be pulled out with a small screwdriver or a knife in order to kill power to the switch. I pulled the air gap tab on the switch. When I pressed it back in, it immediately started doing the same thing.  This was an old first-generation GE Z-wave smart dimmer, which had been installed and working well for over a decade.  I figured it had failed and decided I would just replace it later that evening, since I have spares on-hand.  We ate in the dark. It has been snowing basically continuously for weeks, so after dinner I had to clear snow.  My son likes to run the snow blower, but I still have to be outside with him while he does it. I usually shovel the tighter spots while he clears the driveway and walking paths. The temp is about -8F, so it was every bit as enjoyable as you’d imagine. 

When we came inside, my fingers were swollen and painful from being so cold, but I wanted to get the switch fixed so I could relax for the rest of the evening. The switch, a simple 2-way, is in a 3-gang box for the lights on that side of the kitchen. The switch in question is the center switch in the group. Smart switches are larger and deeper than regular switches, so the wiring in the box is always super tight and a royal pain in the ass to deal with. I was unsure which breaker this particular switch used, so I planned to attach a toner to the wire and identify the breaker by sweeping the breaker panel. I removed the wall plate, but I couldn’t get switch out of the wall far enough to attach a toner without really getting involved with the (live) wires.

Since I needed to determine which breaker it used, and this switch would no longer illuminate the light attached to it, I asked my wife to watch the lights attached to the adjacent switches while I flipped breakers. After flipping one of the breakers, she said all the lights shut off, so I knew I had the right breaker.

With the power cut, I was able to more aggressively pull on the switch and wiring, to get it far enough from the wall to unscrew the terminals. Smart switches, unlike old school switches, have a neutral attached to them. This means there are 4 wires attached to the switch. Most things in my house are wired with wrong colors, so sometimes even the simple things can be a journey of epic discovery. The ground, line, and load wires were removed without incident. While removing the neutral, my screwdriver briefly bridged the line to neutral and POP! I was electrocuted, there was a shower of sparks, and a glowing ember from the remains of the wire landed on the dog bed sitting beneath the switch and burned a hole in it. It also took a chunk out of the shaft of one of my favorite screwdrivers. Outstanding. How the fuck did this happen? Apparently this particular circuit is on a different breaker than the other two switches in the box. I should also mention that I own several non-contact voltage detectors which I affectionately refer to as “chicken sticks.” I usually use them to make sure power is off before messing with stuff like this. I even had one with me, but I was already so frustrated with the process that I skipped testing with it. Lesson learned, don’t trust your wife with electricity.

Naturally, at this point I use the chicken stick to ensure the firework show had tripped the breaker associated with the circuit I am actively working on. It did. Having now confirmed, for the second time, that the circuit had no power, I proceeded to wire up the new switch. I should mention this isn’t actually a new switch. I am installing an Inovelli Z-wave switch that I bought used a while ago, because this lamp doesn’t need anything fancy.

With the switch wired and installed back in the box, it’s now time to pair the switch to the Hubitat hub. Since I have several automations and services tied to this switch, the process of replacing the old switch with the new switch is more complicated than a typical device pairing. I need the new switch to take the place of the ID the original switch used, so that I don’t need to re-bind all of the services and automations associated with it. Z-wave is a now ancient technology that connects smart devices on the 900MHz band. Z-wave devices must be “included” (added to a network) or “excluded” (removed from a network) by a very specific song and dance which varies with every type and model of device. The old GE switch I am replacing, which is the same as most of the switches in my house, uses the same button sequence to enter both include and exclude mode. It enters the include/exclude mode by simply tapping the on or off button on the switch. The Inovelli switch I am replacing it with requires you to tap the auxiliary button rapidly, 3 times. Some devices have a visual indication for the pairing status, and some don’t. If this process is not followed exactly, any number of weird things can happen and you will not succeed. The variance and also particularity of this process is one of the reasons I try to avoid Z-wave devices nowadays.

I decided that even though I’ve grown hate Z-wave, it would be easier to replace this Z-wave switch with another Z-wave switch so that the process could be simplified by using the “replace” feature in the smart home hub. Hubitat’s Z-wave “replace” process, like most Z-wave processes, is a bit finicky. You have to view the Z-wave network details, find the switch in question, initiate a refresh, allow the refresh to fail (sometimes takes multiple attempts), and only at this point will the hub present an option to replace the device. The “replace” option initiates the Z-wave pairing process, but instead of pairing the new switch as a new device it will assume the identity of the failed switch. The process is always a bit of a crap shoot on whether it works properly.

I initiated the Z-wave replace process, and started the inclusion mode on the replacement switch. The hub said the replace was successful, but the switch indicated pairing had failed with a red LED light. Perfect. Since I purchased this switch used, I thought maybe the switch was still paired with another hub and I decided to reset it by performing an exclusion. I put the hub in exclusion mode and initiated an exclude on the switch. This also failed (probably because the switch wasn’t still paired with its original hub). At this point I am frustrated and cursing myself for trying to take the easy route and replace a Z-wave switch with another Z-wave switch. Have I mentioned I fucking hate how much of a pain in the ass Z-wave can be during the pairing process?

On a whim, I started trying to remotely control the switch from the hub. The hub acted like the switch was turning on and off, but nothing was happening to the light. I tried physically pressing the switch to turn the light on and off, and to my surprise the hub showed the light turning on and off. So, the switch was somehow half-paired with the hub. Brilliant. I tried a couple more times to turn the light on and off from the hub, and nothing happened. It was at this point that my wife said, “Is the light in the living room supposed to be turning on and off?” FML. Apparently at the exact moment I initiated the Z-wave “replace” on the hub, she turned that light (and also the ceiling fan) on from the smart switches in the living room, which inadvertently put them in inclusion mode. The hub thought that was the device I was trying to replace, and mapped everything to the living room light. Since the fan switch was also pressed during the process, it somehow messed up the pairing of that switch as well.

At this point, all of the automations tied to the kitchen table lights are now turning the living room lights on and off, and the fan switch is completely catatonic. I cannot turn the fan on or off from the switch or from the hub. When I turn it on, the fan moves about 6″ and the switch immediately turns itself back off. I tried excluding the switch and it had no effect. This has gone well.

I did some more troubleshooting and realized that the replacement switch for the kitchen table paired itself under a new name, and I was able to control the light remotely! I still had to move all the automations that were now bound to the living room light back to the kitchen table light, but that is at least doable. Upon further testing, I noticed that when I press the buttons on the switch, the hub reports that I am “holding” or “double tapping” the buttons, which is not correct. This will mess with some of my automations, so I need to figure it out. At this point I notice some of the other switches in the house are acting weird. Given all the weirdness, I decide to punt and just reboot the hub to see if that helps. When it comes back online, the switches (with the exception of the fan controller) all start behaving normally! I have no idea what happened to get it in that funky state, but at this point I’ll take the win.

But what the fuck happened to the living room fan switch? Why is it dead, even when using the physical buttons? Given my consistently bad luck, I assumed that switch had also coincidentally failed and decided I needed to replace it. I don’t have any spare fan control switches on-hand, so I decided to just reinstall one of the original “dumb” fan control switches for the time being. I am glad I keep all the original switches in a bin in the basement when I replace them with smart switches.

What should’ve been a 10 minute process has turned into an hour and a half, and I just want to be done. I’ve been electrocuted, ruined a fan controller, messed up my screwdriver, burned a hole in the dog bed, and inadvertently mapped a bunch of automations to another switch in the house. Basically a typical Friday night! I rename the new switch and re-map all of the automations to it. I test the dumb fan switch and it works fine. Everything seems ok. Why do I complicate my life with this shit?

My home server uses dual Intel Xeon Gold 6138 processors. They’re getting a bit long in the tooth, so I decided to purchase some used Xeons of a newer generation, with more cores and a higher clock speed. This had all the ingredients to be a huge upgrade, and since I had upgraded the CPUs once before (a few years ago), I was confident it would be a fairly straightforward process.

I chose Xeon Gold 6248s for the upgrade because they have a 25% faster clock speed and only draw marginally more power. To accommodate the CPUs, I needed to purchase upgraded CPU coolers, as the ones I am currently using don’t support 150W TDP CPUs. The eBay reseller I purchased the CPUs from accidentally shipped me 13 CPUs instead of 2, so they ordered UPS to return the shipment to them before it made it to me. They then overnighted me my 2 CPUs once they received the return shipment. So my CPUs made a few trips around the US before they made it to me.

The time comes to finally tear the server apart and begin the upgrade. I slide the server forward on its rails, carefully watching the cables to make sure they don’t get caught on anything. Not carefully enough. I heard a ripping sound and immediately stopped sliding the server. At first, I couldn’t figure out what happened. I saw one of the CAT 6 cables disconnected, but they don’t make tearing/ripping sounds when they are disconnected. Then I saw a small black piece of plastic sitting on top of the UPS (the device racked immediately below the server). It took me a minute to figure out where it came from and then I realized it was the part of the RJ45 port that holds the cable clip in place, to keep the cable seated in the port. It had literally torn the plastic out of the port on the server, so that cables can no longer lock into place in that port. Great. We’re off to an awesome start.

I removed the top of the server, then proceeded to remove the cooling duct that covers the CPUs. Upon removing the CPU coolers and CPUs, I realized I was a bit overzealous with the thermal paste during my last CPU upgrade, so there was some excess paste on the edges of the CPUs that I had to clean up. I got everything cleaned up, and began assembling the new CPUs and CPU coolers. This is when I noticed that the underside of the CPUs were not completely clean. At first I thought some of the contacts were scratched off, but I realized there was just some grease or dirt smeared on the bottom of the CPUs. I used a clean rag to wipe the CPU contacts and they looked much better. I applied thermal paste and installed the CPUs and coolers. The cooling duct had some removable panels that had to be removed in order to accommodate the larger CPU coolers. I buttoned everything back up and powered on the server to see the results. About 10 seconds into the BIOS POST, I received a message relating to the RAM:

This was not encouraging. Since I didn’t touch any of the RAM DIMMs, I thought perhaps the CPU contacts were still dirty and for some reason they weren’t recognizing the RAM. Hoping that no damage was done to the server, I proceeded to power it down and remove the CPUs to inspect them. The contacts were still not perfectly clean, so I decided to use isopropyl alcohol and a paper cloth to clean the contacts. This worked very well and I was happy with the results. Hopeful that this was the issue, I reinstalled everything and powered it back on.

Same error. Shit.

I started googling the scenario and came to a rather disheartening conclusion. My DIMMs are too slow for this generation of CPU so my server is failing the memory test. Shit. Shit. Shit.

My only option at this point is to tuck tail and put the original CPUs back in. I revert everything back to the way it was, and power it back on. All is well, but I had to change my VMware vSwitch to use a different physical port for the network, since I ruined the port I was using.

The money I spent on the CPUs and CPU coolers was completely wasted. I also wasted about 90 minutes, ruined an RJ45 port on my server, and accomplished nothing. What a great way to spend a Tuesday evening!

I ordered a Mopar camshaft over the phone from the local Szott M-59 Jeep dealership.  I provided the part number to the parts rep and he said the part would be here the next day because it was in Cleveland.  I paid for the order with my Amex over the phone because the order could not be placed until I paid, and I was not able to visit the dealership during business hours for the next few days.  I was not informed that this was considered a special order, or that I would not be able to return the part if there was a defect.

The camshaft arrived the next day and I picked it up near the end of the day.  I opened the package the following day and immediately noticed several manufacturing defects on 3 different lobes of the camshaft.  I never attempted to install it because the visible defects would likely cause engine damage.  The reason I ordered the Mopar OEM camshaft is because I thought it would be the best quality camshaft available for my engine.  After seeing the quality of the part I received, I became convinced that Mopar is not the high quality that I thought it was.  I decided then that I wanted to return the part and purchase a different brand of camshaft.

I visited the dealership on Saturday to return the camshaft.  This is when I was first informed that it was considered a special order because they did not have it in stock, and that I could not return it.  Note that this is not a custom part in any way, it is a standard Mopar part.  It just happens to be a part that they apparently don’t stock at this dealership so they had to have it shipped from their distribution warehouse in Cleveland.  I told the parts rep that I was not informed it was considered a special order, nor was I informed that it could not be returned.  He said the information is on my invoice.  I explained that I was not provided my invoice until I picked up the part on Thursday, which was the day after I paid for the part over the phone.  I would not have paid for a part that I couldn’t see if I had been informed it was not returnable.

The parts rep then called the person that took my order over the phone.  They said they could do a warranty replacement of the part.  I said I did not want a replacement part because after seeing the low quality of the part I received, I am going to purchase a camshaft from another manufacturer, and that I wanted to return the part.  At this point he said it could not be returned because I provided them the part number.  I asked what difference it made that I provided the part number?  The part number is correct, and I received the correct part; the part just has manufacturing defects.  I was again informed that the part could not be returned because it was a special order.  I started to get the sense that the rules were being made up as the situation progressed, so I asked to speak with a manager.  The parts rep called the manager on his cell phone, but did not explain much of the situation to them.  The manager also said it could not be returned because it was a special order.  I asked to speak with the manager on speaker, but the manager told the rep he was too busy and didn’t even want to take his call.  I was told to come back next week and speak with the person that took my order.  I asked what would be different about speaking to them in person, when the rep had already called them on the phone?  The rep could not provide an answer.  I then asked to speak with the general manager of the dealership and was informed that they do not work on the weekends and again was told by the parts rep that I could not return the part.  At this point I realized that there was no one of authority at the dealership willing to listen to me or address my concern, so I asked for my camshaft and invoice back, and I left.  

At this point I intended to use Amex return protection to get my refund. When I started that process I realized that for some reason auto parts are not eligible for Amex return protection. Awesome. My only recourse at this point is to go back up to the dealership a different day and hope that one of the managers will actually do their job and listen to my situation instead of hide behind a phone saying they are too busy to help.

Some photos showing the casting defects:

2024 has been an especially hectic year. With Thanksgiving drawing near, and it also being my wife’s favorite holiday, I wanted to try to make it extra special for her this year. My wife does all of the cooking for our family, and she spends a lot of time in the kitchen. When we bought our house, the kitchen had a combination wall oven/microwave installed. We never use the microwave and it would make cooking big meals much easier if we had a double wall oven, so I installed one. My wife is also a big football fan, and the annual Detroit Lions Thanksgiving game is always happening while she is preparing our Thanksgiving feast. I had previously rigged up a laptop in the kitchen so she could watch the game but this year I thought I’d level up and install a TV on the wall in the kitchen.

My mom came over for the day, so she and my son were having fun playing games, looking at pictures, and being entertained by our dog. It was shaping up to be a perfect Thanksgiving! As it got closer to dinner time, my son started getting grumpy. We must’ve asked him 10 times what was bothering him, but he kept saying nothing. He gets like this sometimes, and it is usually fruitless to keep trying to get an answer out of him, so I decided to just let him be grumpy. After working diligently in the kitchen all day, my wife finally sat down so that we could enjoy the meal she had been working hard on all day. As we began to take our first bite, my son says his stomach hurts and he goes to sit on the couch. Not even a minute later, it happened…

From the dining room table, I see my son running toward the bathroom with a creamy substance flowing out of his mouth and onto the floor. My wife runs over to the bathroom to see what was happening, and I followed behind while holding our dog back. It looked like a crime scene. Vomit was probably in 10 different places on the floor in the hallway, and running down the front of the sink, on 3 different walls, on the garbage can, and in, on, and around the toilet. I didn’t know whether to ask if he was OK or call in an exorcism. Then my wife comes out of the bathroom holding a small black object and saying that it probably needs to be thrown away. I looked on in horror, as I realize it’s my Logitech Harmony Elite universal remote control, absolutely covered in vomit! My son had been in the process of turning on the TV when he realized he was going to vomit, and he was in such a hurry to get to the bathroom that he kept the remote in his hand. Obviously I’m more worried about him than the remote, but this remote is one of the best universal remotes of all time, and it went out of production about 5 years ago. Good examples of it are hard to come by, and quite expensive.

After the violence ended, my wife and I began cleaning up the mess. I’d never seen anything like it. The sheer coverage area and variety of affected surfaces was astounding. My son appears to have a gifted ability to spew vomit in a manner that is very difficult to clean. The sections of it that had been sprayed at the front of the sink and toilet had seeped underneath the edges of them. There’s a good chance they’ll need to be removed in order to thoroughly clean underneath each of them. I sense my black Friday will be more brown than black. I would estimate that less than 5% of the content made it into the toilet. My mom kept yelling from the other room offering to help, but I was intent on keeping her out of the bathroom because I didn’t want a slip and fall accident to add to the chaos.

While cleaning the area around the toilet, I managed to spray some cleaner on the leak sensor installed behind the toilet, which sounded the alarm on all of our phones and automatically shut the water off. Perfect for when you’re wrist deep in what is basically sewage, and now have no way to wash your hands!

My mom, now unable to eat and fearing we had the plague, high-tailed it out of the house. When the dust settled, I gazed at the dining room table, which had 4 servings of untouched food, and various serving dishes of food that my wife had been working on all day. None of us got to enjoy the Thanksgiving dinner that we had all been looking forward to. I turned the water back on, washed my hands 10 times, and sat down to eat dinner. No one else was able to eat after that experience, but I’m able to compartmentalize just about anything when I’m hungry. So, I enjoyed Thanksgiving dinner alone, while still somewhat in shock from what I’d just witnessed. This Thanksgiving was definitely not the awesome day I had planned for my wife, but it will definitely go down as one we will not forget!

The TJ started making a faint squeaking noise when going over any kind of bump. Anyone that knows me knows that I can’t stand squeaks and rattles. The noise was coming from the tailgate, because of the oversized spare tire that is mounted to it. The tailgate hinges just aren’t up to the task of supporting such a large tire. There are all kinds of solutions to this problem, but in order to maintain a mostly stock look, I decided to go with some beefed up hinges to stabilize the tailgate.

Installing the new hinges is just 8 bolts, so it should only take a half hour or so, if you include aligning the tailgate. The first 6 bolts came right out, with zero drama. The 7th bolt started to lock up and make loud squeaking noise, which is a tell-tale sign of the threads on the back side of the nut being rusted/corroded. I stopped and hit the bolt with some penetrating oil, and decided to let it sit for a bit. The 8th bolt made similar noises, but came out just fine.

Back to the 7th bolt. The bolt backed out about 1/4″ before it started to lock up. I decided to hit it with more penetrating oil and run it back in, to make sure the threads were lubed. As soon as I tried to tighten it, the bolt snapped with very little effort. FML. I was devastated. I figured my best bet was to drill out most of the bolt and try to remove what’s left with a punch. Naturally, I couldn’t find my center punch or screw extractors, so I took the boy to Home Depot for some supplies.

Home Depot didn’t have a regular center punch, so I had to get one of the stupid automatic ones. I also picked up a set of screw extractors and some cutting oil. I get home and pull the stuff out of the bag and everything is covered in oil. The spout on the cutting oil container was cracked and the oil spilled everywhere inside the bag. Luckily the bag contained it, so it wasn’t all over the car.

After center-punching the bolt to align the bit, I drilled the bolt about 95% of the way to the threads. It was fairly uneventful. The last bit I ran through it actually spun the back half of the bolt out the backside of the nut, so I only had the front half of the bolt left in the nut. I decided to grab the remainder of the bolt with a screw extractor, thinking it likely wasn’t jammed that bad. I got about one full rotation with the screw extractor and SNAP. The screw extractor jammed inside the nut and snapped off.

What you see here is a broken screw extractor inside of a broken bolt, inside of a captive nut inside the body of the TJ, which I removed because of a very minor squeaking sound.

I’m not actually sure how to proceed at this point. Screw extractors are obviously very hard and brittle, so I tried to hit it hard with a punch a few times. I was able to break a few small chunks off of it, but didn’t make much progress at all. I spent some time going at it with a Dremel, but the screw extractor was just too hard for any of the bits I had on hand. I went back up to Home Depot to buy a diamond bit for the Dremel, and I hope to be able to burrow through the center of the screw extractor in the morning. If I can get through the center of it, I hope to be able to crack it into a couple pieces and pull it out.

I hate leaving a project in a state like this, but everything I touched today turned to shit, so it’s time to call it and regroup in the morning.

Day 2

I spent a couple hours trying to make a dent with one (actually, two) of these diamond bits in my Dremel tool. It is an incredibly slow process, and it produces a surprising amount of very fine metal dust given how little material it appears to be removing. It was very difficult to tell where the broken screw extractor ended, and where the nut and surrounding bracket started.

Eventually, at what seemed like glacial pace, I broke through the center of the screw extractor. At this point, I was able to widen the hole with the less worn-down part of the diamond bit. This went much faster than trying to burrow straight through, but it presented a new challenge… I needed to make sure I didn’t go beyond the screw extractor (and the remains of the broken bolt that was still in the hole). If I did, I’d compromise the captive nut in the body, and who knows how the hell I’d recover from that. So, I tried to go slowly and carefully, while looking inside every so often to see if I saw the material change to the bolt or the nut.

Once I could finally see the threads, I broke out some of the remains of the screw extractor with a punch. Now it was just a small portion of the screw extractor and what was left of the bolt. I started trying to run a tap through the hole, but it jammed pretty early on. I wasn’t about to break off a tap in the same damn hole I’d been working on for hours. I kept running the tap up until it jammed, backing it off, adding oil, and running it again. I repeated this process over and over until I started making progress.

Then I heard a crack. My heart sank. I was afraid to move my hands. When I gained the courage to move, I was relieved to learn that the tap was still intact. The crack appeared to come from the screw extractor. I backed the tap out and did what I could to clean out the threads. I ran the top back through the hole and I was able to power through the rest of the junk in the threads and make it completely to the back side of the nut! I backed the tap out and inspected the nut and it appeared damage free! Eureka! Talk about a relief!

I am astounded that I was able to remove the screw extractor and the remains of the broken bolt without ruining the threads on the nut. This felt like a complete impossibility when I ended my day yesterday. Now all that remains is to install the new tailgate bracket.

Thankfully, the new bracket installed drama-free. I’m very happy with the quality and it appears to hold the tailgate much more rigidly. No squeaks, and the tailgate shuts better than it ever has!

I only ordered the hinge kit, because I already have a heavy duty spare tire bracket and I didn’t like the one that was intended to go with these hinges. I expected that I would have to do a little custom work to make my tire bracket work with these hinges. All that remained was to take some measurements so that I could design and 3D print a spacer for the tire bumper pieces that support it.

A few simple measurements and an hour later, I had a nice spacer that wrapped around the new hinge and brings the bumper right back to where it was before the project began. I’d already added some spacers to these bumpers when I installed my heavy duty spare tire bracket; I just needed to make one with a cutout for the new hinge, since it cut into the area the spacer used to occupy.

Victory! I am very happy with the end result, but this is absolutely the longest I’ve ever struggled with a single bolt in my entire life. I’d say it took 5-6 hours of suffering to accomplish what should’ve been a 20-3o minute project.

I had a doctor appointment scheduled for this morning at 8a. It’s cold as hell, so I tried to remote start my truck.

It did not start. No lights, no crank, no nothing. I drove the truck 2 days ago and everything was fine. Something must’ve caused it to drain the battery.

I walk out to my truck to see what is going on and the battery is completely dead. I had to put the key in the door to open it, and was presented with no dome light, etc.

I measure the battery with a voltmeter and it’s zero volts. It’s a 2 year old Optima yellow top deep cycle AGM battery. Most batteries won’t come back from being this dead, but I am determined to try.

I go grab my battery charger to see if I can revive it, but I need to run an extension cord out to the truck to charge it.

The garage door won’t open when I press the button on the wall. The screen on the button says “Press the push bar to activate control” but pressing it does nothing. We had a power event a few days ago, which likely caused something to go crazy with the RATGDO garage door controller I use. It’s times like this where I wonder why I complicate my life with this stuff. I was able to use my phone to open the garage door and drag the cord out to the truck.

I connect the charger to the truck and it doesn’t even recognize it’s connected to a battery. After some screwing around, I manage to get it to engage the 75A engine start function, which wakes up the truck. At this point, the alarm starts going off. HONK HONK HONK. It’s just after 7am. This likely wakes up my son and some of my neighbors. I disconnect the charger, which immediately stops the furious honking, because the battery returned to zero volts. I close everything up and head inside. I will deal with this later when everyone is awake and it’s not dark out.

I call to cancel my doctor appointment and after navigating the phone prompts, I eventually realize there is literally no way to speak to an actual person. Pressing 0 just takes you to the main menu, and pressing 6 for “extra help” just reads off the URL to the website and then hangs up on you. There is also no way to cancel the appointment online, because it is on the same day. About a half hour later I get a nastygram from the doctor because I didn’t show up.

At about 9:30a, one of my meetings ends early so I head back outside to try to get the battery charging. I disconnect the battery from the truck so that the alarm doesn’t go off again.

Now the charger just gives me an error saying there is an open cell in the battery, and it won’t charge. It’s not possible to have an open cell in this battery. I try a different charger. The moment I connect the clamp to the positive terminal, the clamp explodes and part of it hits me in the forehead. Apparently single digit temperatures combined with old plastic and a lot of spring tension equals a spectacular failure of the clamp.

I go get the volt meter to see the state of the battery. The battery in the volt meter is dead. Mind you, this is the same volt meter that I used at 7a, and I did not forget to power it off.

I grab a third battery charger (what, you don’t have 3 battery chargers?). It stops charging almost immediately, saying the battery is “Full.” It reports the battery voltage at 5V.

These “smart” chargers are apparently too smart to charge a battery this dead. I need an old-school charger that just applies 14V to whatever you connect it to, regardless of the circumstances. I’ve probably thrown away a charger like that and now I need it.

After much screwing around, I manage to get the first battery charger to trickle charge the battery, or at least it appears to be. I leave it be, and will come back at lunch and see if I can charge it at a higher rate.

When I check on it at lunch, the battery is measuring 12.1V. The “smart” charger finally agrees to try to charge the battery at 25A. I’m going to leave it like this for the rest of the day and check it around dinner to see what the status is. Chances that it didn’t error out and cancel charging as soon I walked away from it are about 0% today.

I can’t wait to repeat this entire process tomorrow morning, since that is when I rescheduled the blood draw and I have no idea what caused my truck to drain the battery.

This is an ongoing saga that is not yet resolved… However, one thing that has been determined is that Frigidaire has the worst customer service of any company I have ever dealt with in my entire life.

Update: After 35 days, Frigidaire finally replaced the refrigerator, concluding the saga.

9/30: A 3 month old fridge fails on Saturday night (9/30/23). When I went into the kitchen in the morning, there was water all over the floor because the ice in the ice maker had all melted and dripped out of the dispenser on the door.

I call Lowe’s because I have a service contract on it. They say since the fridge is within its 12 month warranty, I have to work with the manufacturer.

Frigidaire service is closed for the weekend.

10/2: Call Frigidaire on monday morning and they say a tech has to come look at the fridge. They assign me to a service company in Flint called Saginaw Valley Service, and say my appointment is Tuesday afternoon

Get a call Monday afternoon from the service company saying they can’t help me because they don’t have any available techs. They say I need to have my case with Frigidaire re-queued to a different service company.

I call Frigidaire and after following the phone prompts I am greeted with a message saying “Due to unusually high call volume, we are unable to take your call.” They end the call. I’ve been trying to get through for the last two hours and haven’t been able to.

I call Lowe’s to see if there’s anything they can do, and now they are unable to find my service contract… the one they successfully looked up on Saturday when I initially called.

I finally get through to Frigidaire again, after 3 hours of trying. They inform me that there are no other service contractors in my area so they have to escalate my case to a “service locator team.” They say this process may take 2-3 business days. I tell them this is unacceptable and I want my fridge replaced, but I get nowhere. The best they can do is “expedite the service locator escalation” so that it takes “only” 24-48 hours to try to find another service contractor. They will not, no matter how much I complain or how many managers (3) I speak with, replace the fridge without waiting for a service appointment.

10/4: More than 48hrs goes by. I call Frigidaire back. They say my case has an update yesterday that says a service company called RSI appliance was assigned to me. They ask if I have gotten a call from them and I tell them no, and that I had no idea a company had been assigned. While on the phone with Frigidaire, they put me on hold to call RSI appliance to see if they can perform the service call. I sit on hold in silence for more than 10 minutes, then magically I hear some hold music. This goes on for another 10-15 mins until someone says “RSI Appliance service, how can I help you?” Frigidaire service somehow blind transferred me to the service contractor, who now needs a reference number that I don’t have. He manages to look me up by the fridge serial number and says the soonest they can come out is October 13th!! I have them book the appointment in case all else fails.

I call Frigidaire back and say I need a service appt sooner than 10/13. They agree that 9 days away is unacceptable and they send my case to the “replacement review team” to process a replacement.

10/5: I have major surgery and cannot deal with fridge drama, so no progress is made.

10/6: Still recovering from surgery, and Frigidaire will be closed tomorrow for the weekend.

10/9: I call Frigidaire on Monday morning to get an update. They rejected the replacement because they found another service contractor (Autumn Appliance), and supposedly scheduled a service appointment. I call Autumn Appliance and no service appointment has been scheduled. The soonest they can come out is 10/16. I have them schedule the appointment, but in my head I am banking on RSI Appliance coming on the 13th.

10/13: RSI appliance calls me to tell me their tech is sick and they won’t be able to make my service appointment today. They reschedule me for 10/20.

10/16: Autumn Appliance visits the house and determines that the Fresh Food evap blower fan has a locked rotor, and the fridge failed diagnostics. The tech sits on the phone with Frigidaire support for over an hour to complete the diagnostics and check the availability of the part. Frigidaire says the fan itself is unavailable so they will need to replace the entire “air tower” in the fridge. Frigidaire verifies that the air tower is available an in stock.

10/17: Autumn Appliance tech calls me to tell me that the air tower is in fact NOT in stock, but now they can order the fan itself. They place the order for the fan and say it will take 3-4 days to arrive and they schedule a service appointment for Tuesday 10/24.

10/23: I call Autumn Appliance to verify that the part came in and they will be able to come to my house tomorrow to service the fridge. The person on the phone says they don’t see that the part came in, but they will check on it and call me back. They call back an hour later and say that the part is back ordered and never shipped. Frigidaire did not tell them it was back ordered when it was ordered the previous week, so the last 8 days have been wasted.

I call Frigidaire to plead my case for a replacement once again. The representative that answers the phone definitely does not speak English as her first language. She can barely understand what I am saying and keep repeating the same thing to me over and over again. At one point she says she will call Autumn Appliance and see if she can get the tech to change the repair order to say the fridge is unrepairable! I inform her that this is absolutely not what I want her to do, and that it should not be up to the service contractor to falsify records in order to try to get Frigidaire to do the right thing. I continue to plead my case and she puts me on hold to “see what she can do.” 1 hour goes by and I am still on hold, with no interaction from the Frigidaire representative. A few minutes later, the line goes dead and hangs up.

10/24: I call Frigidaire to try to get an update. The representative I speak with now says that he needs to escalate the back-order to the service locator team to try to locate the part at a local repair facility/warehouse and that the process will take 5-7 business days. I inform him that time frame is unacceptable and re-explain the history of my case to him. I demand the fridge be replaced. He just continues to repeat the same thing back to me, “We need to complete the service locator process before a replacement can be issued.” He also stated several times that, “If the part cannot be located, we will issue a replacement refrigerator.” I tell him this is the third time I’ve been told that the fridge will be replaced and it still has not been replaced. Eventually, I am able to convince him to assign a “very high priority” to the service locator ticket and he says I will get a response in 24-48 hours. Out of frustration, I accept those terms and end the call.

10/25: On a whim, I look up some Frigidaire (Electrolux) execs on LinkedIn and try to get some contact information from them. With the help of a friend, we’re able to infer the email address of the COO, so I send them a long email that reeks of desperation, in hopes they can help.

10/26: After 48 hours, I did not receive any updates from Frigidaire, so I call them for an update. I was in the middle of a few things today, so I decided to try the text message support for the first time. I was clearly connected to a bot, but I provided my reference number and asked for an update on the backorder escalation. The bot responds that there is no update and she will submit the case to the replacement review team (again). She informs me that it will take 5-7 business days to get an update (again). I respond that 5-7 more business days is unacceptable and I want this to be a high priority request. I go back and forth for several minutes trying to convince the bot that it needs to happen sooner than 5-7 business days, but my efforts were futile. Having already been escalated to the replacement review team earlier in this process, I expect that I will receive no update. When I call next week it will have been rejected and the process will start all over again. This is absolutely maddening.

10/31: I get a call from the appliance service contractor saying that they’re still unable to get the parts needed to repair the fridge. For some reason, they gave me a new case number to track the process.

10/31: I get an email from Frigidaire–the first form of any communication I have received from Frigidaire since this process began–that my case has been closed. No other information is provided.

11/1: I call Frigidaire support to see what happened to my case. They inform me that my replacement has been approved! They hand me off to a local store for replacement logistics. I call the local store and they have no authorization from Frigidaire on the replacement. Outstanding.

11/3: New refrigerator has arrived and seems to work. Finally, this circus is over!

I needed to replace the water pump and the lifters on my Jeep TJ. The lifters are quite involved, since you have to remove the head from the engine. I decided to do this job in the winter, since I don’t typically drive the Jeep in the winter and it wouldn’t matter how long it was torn apart. Boy am I glad I did.

While the engine was torn down, I decided to replace a bunch of other maintenance items: radiator, radiator hoses, heater hoses, and the timing set (chain and gears). I had ordered some of the parts back in the summer, so that I knew I would have what I needed on-hand. This summer when I replaced the oil pan gasket, I could access the timing chain and I noticed it had quite a lot of slop in it. They are known to last upwards of 300k miles in this engine, but it was easy enough to replace it while I had everything else torn apart. Or so I thought.

My good friend Joel was a valuable second set of hands for the entire project. We encountered some minor difficulties with the other items on the list. For instance, I could not get the lifters out with a magnet because they were stuck in their bores. I scrambled and bought a tool to get them out, but the tool was garbage. Thankfully, my neighbor had a much better version of the tool and let me use it. We got the head reinstalled and were making great progress. I decided to tackle the timing set last, as I expected it to be fairly straightforward. After all, it is just two gears and a chain.

The old timing set came off without incident. I grabbed the new set and slid the small sprocket on the crankshaft. It got stuck and would not slide on.

You can see in this photo that the crankshaft has two keys on it. The outermost key is for the harmonic balancer, and the inner key is for the timing gear. The new gear was binding on the inner key and would not slide back against the engine block. I took the gear off and inspected it. No burrs or other crud in the keyway that would prevent it from fitting. I inspected the key on the crankshaft with the same result. I took the old gear and slid it right back on the crankshaft without any resistance. I compared the two gears and they looked nearly identical, but clearly the keyway on the new gear was narrower. I grabbed a set of digital calipers and measured the keyways. This was more difficult than I thought, because the shape/size of the keyway makes it very difficult to get calipers inside for measurement.

The original gear keyway measured about 0.1890″.
The new keyway measured about 0.1850″.
The crankshaft key measured 0.1880″.
The new gear was clearly too small. I called Cloyes (the manufacturer of the timing set). They were able to give me the spec for that keyway, which was 0.1885 – 0.1910″. I grabbed a file and tried to open up the keyway a bit. The gear appeared very hard, and the file didn’t seem to have any effect, so I stopped. I called Cloyes back and let them know that my gear was out of spec. The tech had someone measure the gears they had on-hand and they were all also undersized and out of spec. They didn’t have one gear that was within spec to ship to me. Since I bought this timing set back in August, there was no way I could return it now (January). I decided to try to file it again.

Out of the arsenal of files I have in the toolbox, only about 5 of them fit inside the keyway and were aggressive enough to attempt to remove any of the steel. I spent about an hour with the gear in a vice, putting a lot of weight on the files trying to remove any material at all. I managed to get the keyway to open up to about 0.1855 or 0.1860″, but that was not enough. Frustrated, I began my search for a replacement.

I wanted to find another brand, to try to avoid getting another out of spec gear. I called every local auto parts store, and all of them only carried Cloyes timing parts for the Jeep. I went on Rock Auto and ordered a Melling timing set, and another Cloyes sprocket (just the sprocket was pretty cheap) just to see if I could get lucky. If the new sprocket fit, I could return the Melling set and save some money. They arrived a couple days later. The Cloyes sprocket was identical to what I had, and measured the same, so it did not fit. No surprise there. I opened the Melling set, and to my surprise there was a Cloyes set inside the box. Apparently Melling just rebrands Cloyes products in this case. This gear was also out of spec and did not fit. For those keeping score at home, I now have 3 identical Cloyes timing gears, and none of them fit.

Frustrated, I began another online search for alternative brands of timing sets for this engine. Most of the sets I could find were for the earlier model of this engine with a different type of camshaft sprocket (the larger sprocket). You are supposed to install these as a set, and not mix and match new and old gears or gears from different sets, as the gears and chain wear together. I eventually located a Comp Cams timing set on Summit Racing, and ordered it. I paid for overnight shipping because I just want to be done with it at this point.

The Comp Cams timing set arrived the next morning, and I could not resist a quick test fit. The gear slid right on the crankshaft, with virtually no resistance. Victory!

The Comp Cams sprockets are cast instead of milled like the Cloyes unit. They definitely aren’t as nice, but they fit and I’m rolling with it!

A few weeks ago, I noticed that my ZFS array was resilvering, due to a HD failure.  This is the first time a drive has failed in my ZFS array, which is a little over 2 years old.  My ZFS pool was 99.7% full; an issue I’ve been meaning to deal with for quite some time now, but have had other priorities.  As a result, a resilver (rebuild/resync in RAID terms) is causing quite a bit of thrashing on the disks.

A bit of background:  In early 2011, I built the successor to my 20HD 30TB Raid6 array.  It is a 24HD 48TB ZFS RaidZ2 array.  RaidZ2 is similar to Raid6, in that two drives are used for parity rather than storage.  This means that two drives can fail without losing any data.  I knowingly went outside best practices while building it, and put 24 drives in one vdev.  Actually 23 drives in the vdev, and one hot spare.  A vdev is considered a group of drives within a ZFS pool.  Parity is localized to a vdev, and you can have multiple vdevs within a pool (with more parity drives in each vdev).  You are only supposed to have up to 9 drives in a RaidZ2 vdev.  At the time, it seemed like the only disadvantage to having that many drives in a pool was performance, which was not a huge factor for me.  ZFS doesn’t stripe across drives in a vdev, so your vdev is only as fast as your slowest drive.  What I didn’t consider was the amount of stress on the drives, and overall time it would take for a resilver to complete.  Especially when your pool is 99.7% full, and it has to move data around in little tiny chunks.

So back to the drive failure.  This should be no big deal.  I had a hot spare, which is why the array was resilvering itself, without any intervention from me.  By the time I noticed it, the resilver was about 3 hours in.  The ETA to complete the resilver was 72 hours from then!  That’s 72 hours of continuous hard disk thrashing, in addition to the normal load caused by the r/w of the 22 VMs I have running on that array.  I shut some of my VMs down, to hopefully speed up the process, and checked back in a few hours.  To my horror and dismay, the hot spare failed, and 4 other drives had taken IO errors (and were being resilvered as a result).  The array continued to resilver, across the remaining drives, and was still thrashing like hell.  Several hours later, 3 more drives had taken IO errors.  That’s 8 drives resilvering, and 2 faulted in the pool.  It doesn’t get much worse than this.  I always buy my drives from multiple sources when I build a NAS, so that they will have different dates of manufacture, and are less likely to fail in huge batches.  What the hell was going on?  All I could figure was that the incredible stress of the resilver was too much for my consumer-grade HDs to handle.

About 10pm that night, it happened.  A third drive failed, less than 18 hours into the resilver.  Since one of the three was the hot spare, technically only two drives from the original pool had failed, which is the maximum allowed without losing data.  At this point I am shitting bricks, and literally can’t sleep.  I shut all my VMs down, and am scrambling to move critical data to other disks in the house.  I lit up my old array, which hadn’t been powered on since we moved into the new house.  It wouldn’t boot!  Something was up with the OS on the boot drive, so I booted off of a Ubuntu Live CD and mounted the array.  All was fine, but the data (which was originally a backup of what was on the new array) was quite stale.  Since 48TB > 30TB, I obviously had to decide what I was willing to lose, and only copy some stuff over.  I started using external USB drives, and my desktop machine as temporary storage to move data to, in case another drive failed.  The next morning my wife says, “Dave, is something supposed to be beeping in the furnace room?”  This can only mean one thing.  A drive failed on my old array (which has an enterprise RAID card on it, and notifies you when a drive fails).  What else could go wrong?  Since my VMs were shut down, I did not get an email notification.  I hopped on the console and noticed that drive 9 had failed, and the array was rebuilding with drive 10 (the hot spare).  The ETA on this rebuild was much less: 10 hours.  It completes without incident, and I swap the bad HD with a cold spare that I had on-hand.

Eventually, the resilver completes, in 73hrs.  No more drives failed, and I haven’t lost any data.  I’m relieved, but still incredibly spooked that I could lose it all at any minute, if another drive failed.  Throughout all of this, I’ve been trying to figure out what my long-term plan was going to be.  Up until this all happened, I had been considering rebuilding my old array (the one with the hardware RAID card in it) with larger (and more) disks.  But now there is critical data copied on that array, and I can’t scrap it and start over.  It seemed like my only option was to build another (third) NAS server, at a considerable expense.  I could go ZFS, which requires lots of RAM (expensive), or RAID, which requires a hardware RAID card (expensive).  I’m highly annoyed, because if I had just addressed this a few months ago when I knew I was running out of space, I would not have to build a third server.  Then it occurred to me that I could buy a drive enclosure with a built-in SAS expander, and connect it to my existing server.  That would require me to upgrade the amount of RAM (ZFS likes RAM), but it was doable.  Of course, I would have to scrap my existing RAM, because it was ECC unbuffered, and I had maxed out what my motherboard could handle (48GB).  I would have to purchase ECC Registered DIMMS to go beyond the 48GB barrier.  I telling my tale of woes to a coworker, and he mentioned that we had a bunch of servers in our warehouse that we weren’t using, and that he thought they were full of RAM.  I checked it out, and they were indeed full of RAM.  352GB of ECC Registered RAM, to be exact!  So I borrowed 12 8GB DIMMs and put them in my server.  Voila!  96GB!

I ordered up my enclosure, a Norco DS-24E, and 8 Toshiba 3TB 7200RPM SATA drives.  I figured I would start with 8 drives, and expand later on.  The enclosure and drives arrived a few days later, and appeared to install without incident.  That is, until I realized that all of the drives detected as 2.2TB drives.  WTF?  Some googling quickly revealed that the LSI SAS1068E chipsets on my SAS controllers did not support 3TB drives!  At this point it’s been over a week since the first drive failed, and I’m on borrowed time with this array.  After a few hours of research, I order a LSI SAS2008 PCI-E SAS HBA.  It’s not the best, or newest, but it is known to work in the very unique configuration I am running (ESXi hardware passthrough to a Solaris VM, to share the array back to ESXi via NFS).  I also ordered 8 more 3TB drives, because I realized that because of my only putting 8 drives in a vdev now, I will have much less usable space and still needed more room to temporarily store my data.  This is getting quite expensive!

The new controller and drives show up 2 days later, and I begin surgery.  It goes surprisingly well.  ZFS is fantastically resilient and scalable.  After an export/import, the pool detected perfectly on the new controller, even though all of the drive IDs had changed.  I was super relieved at this point.  I then added the other 8 drives to the enclosure, and built the new pool as 2 8 drive RaidZ2 vdevs.  The pool created without incident.  I enabled compression, nfs and smb on the pool, and immediately began copying my data to it.  It’s now been a little over 24 hours, and 20T worth of the data has been copied to it.  I intend to get a current copy of everything on the failing array, and scrap it completely.  I will then rebuild it with 3 8 drive RaidZ2 vdevs, just like the new array, and forgo the hot spare.  I’ll lose a significant amount of storage (6TB), but this whole event will be much less likely to occur again.  Any future resilvering will be limited to 8 drives, instead of 24.  Also, my IOPS will be greatly improved, because ZFS stripes across multiple vdevs.

So the night before thanksgiving, I’m doing some work in my office and I hear, “Dave…  The TV isn’t working.”  Great.  The DirecTV HD DVR in my living room evidently crapped out.  It locked up, and upon rebooting it I was greeted with a “ERROR 14 – Internal Storage Problem.”  Several reboots later, I was on the phone with DirecTV.  They ordered a replacement receiver, but could not guarantee what model I would receive.  Over a year ago, I fought tooth and nail to get the HR24 receiver, because the previous models are painfully slow.  I was certain that I was going to get one of the previous models, because the guy said I would be receiving a refurbished unit. Oh, and did I mention I have to pay $20 for shipping on the replacement receiver?

The order didn’t get officially entered until Friday afternoon, due to the holiday.  7 days go by…

Last night I received the replacement unit.  To my surprise, it was an HR24!  I was quite pleased.  I immediately unboxed it and hooked it up in place of the dead unit.  Upon plugging it in, I kept hearing the “bong” noise through the speakers that you normally hear when you press an invalid button on the remote.  My first thought was that they had shipped me a remote and I didn’t see it in the box, and the buttons were being mashed.  No remote in the box.  I went through the entire activation process (muted the audio).  When the programming finally came up, it started wildly cycling through the screen resolutions.  So fast that my TV couldn’t keep up and I basically just saw a black screen.  The resolution indicator on the front of the unit was rapidly and erratically cycling between 480p, 720p, 1080i, 1080p.  This particular unit has a touch-sensitive front panel, so it does not have any physical “hard” buttons.  For some reason, it thinks the resolution button is being pressed repeatedly.  In an effort to stop it, I press the resolution “button”, and some of the other buttons on the front of the unit.  No joy.  It just keeps cycling through the resolutions.  I thought maybe I should block the front of the unit, to make sure that it wasn’t getting some erroneous IR signal causing this to happen.  The cycling continued.  So again, I’m on the phone with DirecTV.  This time they run me through 20 minutes of useless troubleshooting.  “Try unplugging the HDMI cable from the receiver.  Ok, reboot it.  Ok, now try unplugging the HDMI cable from the TV.”  ?!  Eventually, the “tech” concludes that the replacement receiver has an internal issue.  They assign me a case manager and tell me that I will be contacted within two hours.  Mind you, it is 8:30pm on a Wednesday.  Unsurprisingly, I don’t get a phone call that night.

This morning I got a call just after 8am from a case manager in Colorado.  She asked me to explain to her, in detail, the issue and the steps I’ve taken to try to resolve it.  She recommends having a technician visit my house.  I tell her that I don’t want to take time off work to resolve this issue.  She says that my only other option is “chance” having another receiver shipped to my house.  She says that she has no way to guarantee that it will be an HR24, and that it may or may not have a similar issue.  How can you run a business like this?  Is there no quality control?  By the tone in her voice, I could tell that she did not have a high level of confidence in the refurbished receivers that ship from their warehouse.  I suspect that their quality control process consists of little more than checking to see if the lights come on when it is plugged in, and then it gets shipped out to another poor sap.  There is no shipping charge for the second replacement, and I managed to get her to credit me the shipping for the first unit.  Still not really compensation for not having service for what will probably wind up being 11 days (at best).

I’m amazed by how little control anyone seems to have over this process.  She said the receiver should arrive in 2 days, but she couldn’t promise that the order would get processed today.  It was 8:30am (6:30am her time) on a week day.  If she submitted the order during the call, how could it not be processed the same day?  She had no ability to choose to ship me a new receiver, vs a refurbished one.  Apparently the only way you can get a new receiver is to have a tech visit your home and install it himself.  What a sweet setup.

I realize it’s just TV.  Big fucking deal, in the grand scheme of things.  But this kind of stuff infuriates me to no end!  So, DirecTV, you are on very thin ice.  One more strike and I’ll be choosing an alternative content provider.

To be continued…