Data Cleaning

About a year ago, the other half had a computer that malfunctioned in a particularly nasty way.

All of the data from that computer was essentially wiped because the computer itself managed to rewrite data enough times on an SSD that it killed the drive. FYI. Solid State Drives have a finite lifespan that is dictated by the number of times data is written to them.

The other half’s computer managed to kill not one but two SSD drives in 6 months. Wow! that’s a whole lot of rewriting of data. 20 years worth in 3 months.

The upshot of all of this is that I ended up digging data up from backups, and places on the network where they’d stored data. This doesn’t sound like much of an issue until you realize that the other half stores data in a completely random way.

This is not surprising since hardcopy data is stored in exactly the same way in odd little places all around the house.

After I’d consolidated all the data I could recover into a single group of folders on the server, I said, “Okay that’s all I can do, you’re going to have to sort through and delete duplicates and what you don’t need.”

That was over a year ago. Guess what? Nothing has been touched since the important documents got transferred to the new computer.

Here’s a lovely chart showing how much data is needlessly occupying space on the server.

These numbers are astounding. Gigabytes of duplicate data? Really?

Well, since none of this has been touched in over a year. I’m going to clean house!

This is going to take a really long time. I’m not even going to bother to examine files, I’m going to use a utility to merge folders and delete duplications.

Whatever is lost will probably be of no consequence since it wasn’t important enough to look through in the first place.

The fact is, there’s no reason to have 5 folders of duplicate information indexed and stored on the server. It’s not that I’m all that worried about space I’ve got tons available. It’s about the possibility of a drive crash and all of this crap would make recovering from a one or two drive malfunction in the RAID array really tedious.

I’m sure that I’ll hear about something being lost, sometime in the future, but I’ll deal with that when/if the time comes.

At this point, I’m curious to see just how the deletion of all of this crap affects the performance of the server. I’m betting that the reduction in size of the index files alone will get me a speed boost.

I just hope the utility can swallow and digest a mountain of redundant data.

I should know in another 12 hours or so.

Now off to deal with some other poop. This time it’s legitimate poo, from the dog. Strangely I don’t mind cleaning the poop from the back yard nearly as much as I mind cleaning up the poop on the server.

Have a great weekend.


Update:

260GB deleted. Probably 100GB more to sort through.

The cleaning application, got lost in the mess. This happened several times.

I finally got annoyed with the utility. I was seriously considering asking for a refund. I went in manually to do the job. Initially I was thinking, “DAMN a human brain will still be needed to do the most basic things!”

Uh Huh…

I found that the application was having a problem with folders nested within folders that all had the same names.

It looked like the image to the right.

But in each one of the subfolders, there were files. Some of these files were originals and some of them were duplicates.

This diagram shows a recursion only 5 levels deep. Something like 63 directories and what I was looking at was in some cases 7 levels deep. It was not uncommon for there to be 1000’s of files in each directory.

As you can imagine, it got out of hand very quickly. Especially when you consider that many of the file names were duplications.

I’d been annoyed at the utility for not being able to keep it all straight. But I was far less annoyed when I realized that I, (The Human,) had gotten lost more than once in this digital house of mirrors.

As a interesting aside, It took me two different applications to create even this simple representation. One of the applications flat out refused to engage in the illogic and crashed.

The second application was somewhat uncooperative but eventually allowed me to create it, then save it as a PNG.

I’m no longer annoyed at the duplicate finding utility that threw up its hands. Even trying to create the representation was harder than I thought it would be. I could picture in my head what I wanted to show, but translating that kind of irrationality to something clear was… Odd.

In the case of the utility, I very much doubt that the programmer who created it even considered that someone would do something like this. So I still think the utility was worth what I paid for it.

Looking at the recycle bin, it looks as though the utility handled 4 levels of recursion without a complaint.

It’s because of this kind of thing that I personally am a bastard when it comes to training computer users on the importance of reasonable directory structures. I’ve always said that if you need nested directories more than five levels deep, you’re doing something wrong.

Obviously, there are exceptions to this rule. Computers can keep track of much deeper nesting, but non-technical average human beings??? More than five levels down, you’re asking for files to be “lost” and setting yourself up to chew hard disk space with duplicate files that will never be accessed again.

Yet again… There’s usually a good reason for me doing things the way I do.

So that was my Saturday… I hope yours was more fun and less, uhh… interesting.

I’m just not that into it…

There was a time in the recent past when I, like many other Apple users waited on pins & needles for the next iOS, iPhone, iPad, MacOS or MacBook computer.

This last iPhone event left me feeling sort of meh.

iOS 15 is somewhat interesting but I’m in no great hurry to update. This is very unlike previous years.

The iPhone 13 Pro is nice, but not dazzling. I sorta like the pale ice blue color but since I use a case that isn’t really all that compelling. My iPhone 12 Pro is still quite serviceable and does what I want it to do.

I’ve currently got an Apple Watch series 5.

I thought that I’d upgrade to the watch 7. I originally got an Apple Watch Series 3, then upgraded to the 5 and thought a 2 year upgrade cycle would probably be reasonable. Looking at the Series 7, I see no compelling reason to upgrade. Okay, it’s got a bigger screen and what else does it get me?

Apple’s presentation was even Ho Hum, it was almost like Apple themselves was “phoning it in“. Every Apple event they’re gushing about how impressed they are with their own technology. How each product, “…is the most advanced Technology Apple has created…”

Every time they say that now, I fight not to say, “Duh” out loud.

I’m always way more interested when they compare their technology to the competition.

I find myself wondering why I wasn’t excited, or even disappointed with Apple this year. I’d have expected to feel something one way or the other, I just don’t.

Early iOS adoption statistics might be indicating that I may not be alone. I caught an article last week, (which I can’t find now,) that said the one week adoption rate for iOS 15 was substantially lower than for iOS 14 in the same period last year.

Why have I not updated? That’s simple, I dealt with bugs that messed my phone up on both iOS 13 & iOS 14. I decided that this time around I’d wait. But when you add the whole issue of the CSAM scanning and privacy issues that honestly feel like an about face for Apple…

Well, I guess it feels a bit like betrayal.


It’s akin to that feeling you have finding out your spouse is sleeping with someone else. When you find something like that out, you tend to get angry, then when you calm down, you start evaluating the choices available to you.

Divorce is messy and ends up hurting you as much as your spouse. That’s kind of the nuclear option.

Ignoring it and hoping it doesn’t happen again, is another option. Sometimes that works, but can you ever really trust them again? Do you find yourself questioning everything they say or do for the rest of your life?

Couples counseling is another option. But that only works if both parties are willing to realize that a) they’re in trouble, b) the relationship still means something, c) are willing to give it a go and mend fences.

To some extent, Apple has behaved like the spouse that was caught screwing around. They’ve quietly admitted that CSAM requires more work. Their decision to hold off on rolling it out feels like, “I’m sorry I got caught. I promise I won’t do that again.

No spouse, ever totally believes that. The one saying it, knows that there will be an opportunity in the future that simply can’t be ignored. The one hearing it, knows this too.

(I’m specifically being gender neutral above, because I’ve seen women cheating on their husbands and men cheating on their wives. Interestingly, I’ve been the third wheel in both equations. Not really proud of that, it’s life, it just happens sometimes.)


I think right now, I’m personally at the “Ignoring it phase,” with Apple. I would consider the “Couples counseling,” phase. But I’m wondering if divorce from Apple is in the cards.

I have to wonder if this is the way a lot of Apple users are feeling. The article I mentioned, was trying to cover the lower adoption numbers by highlighting that unlike previous years Apple is going to be supporting iOS 14 and iOS 15 bug fixes concurrently. In the past when Apple released a new iOS they stopped development on the previous version.

This meant that there were no security patches or bug fixes for the older version and if you wanted to close those security holes, you had no choice but to upgrade the OS. This year is different. Apple has said they’ll continue the security patches indefinitely.

I’m sure that some of the upgrade hesitancy is due to this, but I seriously doubt that all of it can be explained away. This feels more like “coming home from somewhere you never should have been,” (Thank you Garth Brooks) and finding your pillow and some ratty blanket on the couch.

I’ll admit that pillow on the couch is way better than a confrontation in the driveway and a gun shot under cover of thunder in the distance. But it’s just as much a statement. (Yeah, I’ve gotten the pillow once or twice too.)

Right now, I’m giving Apple the pillow. We’ll have to see just how contrite they are, and how willing they are to keep my business going forward.

Gee Thanks Joe!

So as has been stated, I’ve been looking for a job for 2 years. I was laid off from my previous job due to outsourcing in Aug of 2019. Just in time for COVID Yea!

Throughout 2020 I applied for various jobs in my field that process has continued into 2021, with no positive results.

With over 30 years experience in technology and software testing I’m apparently unemployable. That’s not whining, it’s just a statement of fact.

Way back in the day, I demonstrated that I had a high aptitude for technology and computers. I was literally hired out of a junior college and never went back to finish a degree. That’s on me I admit it.

The fact is most of what was being taught in colleges had zero relevance to what I was actually doing because the colleges were teaching technology that was already 5 -10 years out of date and I consistently found myself on the bleeding edge of new technology.

There was little incentive or indeed value to paying for knowledge that was irrelevant. You see, my pragmatism dictates that knowledge is useful, a piece of paper proving my indebtedness, less so.

New hardware, new software, new languages, new technology was coming out every month. It was all most of us could to just to keep up within the companies we were working for.

While all this was going on, there were the innumerable layoffs, mergers, and acquisitions that made someone like me have a resume that looked like I couldn’t keep a job.

That assumption is completely false, but as time moved on and younger HR people came into the industry, they were applying all their college knowledge to my resume and frankly there was a significant disconnect.

Over time it got harder and harder to find a job, but I persevered and remained gainfully employed.

In my career, I have been a technician (back when we actually repaired machines at a board level), I’ve been a technical instructor (teaching others how machines worked and how to repair them), I’ve been a regional representative (supporting corporate product sales, and solving problems of product implementation and repair), I’ve been a technical support representative (explaining very technical issues to non-technical people to solve their problems), I’ve also worked in retail and warehousing.

All of this experience is useless today because I don’t have a degree. I ask you what function a computer science degree would serve since that degree would include FORTRAN-77 or COBOL, parallel communication, or RS-232? Virtually none of these skills have any relevance today.

To be sure, subsequent technologies that grew out of the aforementioned do have relevance and those technologies would have been learned on the job over the intervening years.

Which is to say, I’m on an equal footing with anybody coming out of college today with the possible exception of those educated in C# or some of the later languages. That being said, the core logic of computer languages is still the same. High level languages compile down to an instruction set that commands the processor to carry out specific actions. That hasn’t changed since computers occupied entire buildings.

The language simply provides a more human readable and therefore easier mechanism to create software. You can still bypass all of those higher languages and write software directly in assembly code. (The instruction set specific each processor. Before you ask, there are still people who earn really good money doing exactly that. Those folks, I respect immensely. Coding at that level is tedious and abstract beyond belief. I’m a little too ADD, or not ADD enough to do anything more than ‘tinker’ in that realm.)

As an aside, the folks that code at that level, are the folks ALL the sexy high level languages rely on. Assembly coders are an interesting and unique bunch. It’s not unusual for them to be socially challenged and challenging to more “normal” people. Assembly coders, deal in absolutes, mathematics, and the purity of silicon switching. Right and Wrong are terms which have no grey areas 2+2 in their world always equals 4. Something works and is therefore “right” or it doesn’t and is therefore “wrong”.

I haven’t met any coders who work in the new field of quantum computing. I would guess that they are a completely different kind of duck. Quantum theory being somewhat less determinant and the underlying math is so far beyond me I can’t begin to visualize it. I’m quite content to take it on faith that they know what they’re doing.

I digress…

The point is, I actually like computers and technology. I like testing it, and verifying that the code does what it’s supposed to do. It’s a puzzle to me. My job in the last decade or two has been to test the software coders create. I’ve mostly looked at this as a cooperative effort where I’m a fresh set of eyes working on code that the actual programmer finished working on weeks before. They’ve moved on to other parts of the project and are being productive. I’m making sure they didn’t miss anything and if they did, I’m the one who captures the errant behavior and shows them how I induced the error.

In this capacity, I’ve learned so much and been blessed to work with some truly amazing people. I earned the nickname “Demon” and wore it proudly. My “Demonic” tests, helped to produce a wide variety of award winning and useful products, of which I am also proud.

Now I’m dealing with having all that taken from me. It was hard enough to deal with human HR folks and get myself, or my resume in front of a hiring manager. Most hiring managers look at my resume and think, “This guy has been around. I’ll bet he knows all kinds of things that I could leverage.” That’s how I’d get hired.

But that was when humans actually read a resume. Now, folks like me are lost in the filters of HR databases. Databases I might add, that are controlled exclusively by the priesthood contained in the bureaucracy of layers of HR.

The hiring manager writes out a list of ideas about the kind of person they’d like to fill a position. HR passes the requirements up level after level of representatives ultimately getting the request to one or two people who actually enter the requirements into a database. However, by the time the data is actually presented to the hiring system the requirements are absolute and glacial.

The hiring manager wrote the initial requirements in a fuzzy way. “This or this would work for the position.” HR enters the requirements as “THIS AND THIS MUST BE PRESENT.”

(C) Scott Adams 2008

That’s how you end up with tons of people out of work, and corporate HR saying there’s no-one available for a particular requisition. This is also how you end up with corporations not capitalizing on workers within their own ranks for internal openings that could be promotions, and why so many workers leave a company after a few years.

I’ve personally witnessed newly minted Baccalaureates applying for open positions within the company they were already working for and being ignored. After 6 months or so they’re tired of being passed over and they go to work for the competition in the position their new degree qualifies them for.

Then I’ve also gotten nasty looks from HR when I laughed in their faces as they lamented that workers had no loyalty and how difficult it was to ramp new hires up to being productive. When asked what I find so funny, I’ve told them exactly why people in my department left. It wasn’t the pay, it wasn’t anything other than HR locking them into a particular position and having no hope of advancement.

As a side note HR people really hate having their noses rubbed in their own poo.

All of this is what workers deal with every day. This is one of the reasons the hiring process is such a royal pain in the ass, for employers and prospective employees.

But along comes Joe Biden… With his imperial decrees about mandated vaccines, and what do we have?

Now we have companies adding “Must Be Vaccinated” to their job openings.

And in HR’s usual moronic fashion, they’re making the vaccine mandate apply even to people who are applying for remote only positions.

So now there’s another hurdle to surmount. It’s not enough that the remote position is on the other side of the country and a prospective employee will never darken their corporate door. A remote only worker cannot transfer any disease to any other employees via a zoom teleconference. There is zero risk or threat of contamination.

Yet, corporations will demand that a worker be vaccinated simply because they can. Their HR departments will fall back on the excuse, “We’re in compliance with The President’s mandate, and the law. After all it’s for your protection…”

Translated: “We’re protecting the corporation from any/all liability. We’re doing our bureaucratic duty…”

So thanks President Biden, you’re batting 1000 at screwing everyone. Good job!

(AP Photo/Charlie Neibergall)

The trouble with career politicians is that they’ve never had to consider unintended consequences. That fact coupled with greed and elitism is why politicians always fail.

If good ol Joe keeps up the pace of failure, I suspect that Washington DC will very soon look like The Vatican. The question will be, are the walls keeping the angry population out, or the shitty politicians in?