Data Cleaning

About a year ago, the other half had a computer that malfunctioned in a particularly nasty way.

All of the data from that computer was essentially wiped because the computer itself managed to rewrite data enough times on an SSD that it killed the drive. FYI. Solid State Drives have a finite lifespan that is dictated by the number of times data is written to them.

The other half’s computer managed to kill not one but two SSD drives in 6 months. Wow! that’s a whole lot of rewriting of data. 20 years worth in 3 months.

The upshot of all of this is that I ended up digging data up from backups, and places on the network where they’d stored data. This doesn’t sound like much of an issue until you realize that the other half stores data in a completely random way.

This is not surprising since hardcopy data is stored in exactly the same way in odd little places all around the house.

After I’d consolidated all the data I could recover into a single group of folders on the server, I said, “Okay that’s all I can do, you’re going to have to sort through and delete duplicates and what you don’t need.”

That was over a year ago. Guess what? Nothing has been touched since the important documents got transferred to the new computer.

Here’s a lovely chart showing how much data is needlessly occupying space on the server.

These numbers are astounding. Gigabytes of duplicate data? Really?

Well, since none of this has been touched in over a year. I’m going to clean house!

This is going to take a really long time. I’m not even going to bother to examine files, I’m going to use a utility to merge folders and delete duplications.

Whatever is lost will probably be of no consequence since it wasn’t important enough to look through in the first place.

The fact is, there’s no reason to have 5 folders of duplicate information indexed and stored on the server. It’s not that I’m all that worried about space I’ve got tons available. It’s about the possibility of a drive crash and all of this crap would make recovering from a one or two drive malfunction in the RAID array really tedious.

I’m sure that I’ll hear about something being lost, sometime in the future, but I’ll deal with that when/if the time comes.

At this point, I’m curious to see just how the deletion of all of this crap affects the performance of the server. I’m betting that the reduction in size of the index files alone will get me a speed boost.

I just hope the utility can swallow and digest a mountain of redundant data.

I should know in another 12 hours or so.

Now off to deal with some other poop. This time it’s legitimate poo, from the dog. Strangely I don’t mind cleaning the poop from the back yard nearly as much as I mind cleaning up the poop on the server.

Have a great weekend.


Update:

260GB deleted. Probably 100GB more to sort through.

The cleaning application, got lost in the mess. This happened several times.

I finally got annoyed with the utility. I was seriously considering asking for a refund. I went in manually to do the job. Initially I was thinking, “DAMN a human brain will still be needed to do the most basic things!”

Uh Huh…

I found that the application was having a problem with folders nested within folders that all had the same names.

It looked like the image to the right.

But in each one of the subfolders, there were files. Some of these files were originals and some of them were duplicates.

This diagram shows a recursion only 5 levels deep. Something like 63 directories and what I was looking at was in some cases 7 levels deep. It was not uncommon for there to be 1000’s of files in each directory.

As you can imagine, it got out of hand very quickly. Especially when you consider that many of the file names were duplications.

I’d been annoyed at the utility for not being able to keep it all straight. But I was far less annoyed when I realized that I, (The Human,) had gotten lost more than once in this digital house of mirrors.

As a interesting aside, It took me two different applications to create even this simple representation. One of the applications flat out refused to engage in the illogic and crashed.

The second application was somewhat uncooperative but eventually allowed me to create it, then save it as a PNG.

I’m no longer annoyed at the duplicate finding utility that threw up its hands. Even trying to create the representation was harder than I thought it would be. I could picture in my head what I wanted to show, but translating that kind of irrationality to something clear was… Odd.

In the case of the utility, I very much doubt that the programmer who created it even considered that someone would do something like this. So I still think the utility was worth what I paid for it.

Looking at the recycle bin, it looks as though the utility handled 4 levels of recursion without a complaint.

It’s because of this kind of thing that I personally am a bastard when it comes to training computer users on the importance of reasonable directory structures. I’ve always said that if you need nested directories more than five levels deep, you’re doing something wrong.

Obviously, there are exceptions to this rule. Computers can keep track of much deeper nesting, but non-technical average human beings??? More than five levels down, you’re asking for files to be “lost” and setting yourself up to chew hard disk space with duplicate files that will never be accessed again.

Yet again… There’s usually a good reason for me doing things the way I do.

So that was my Saturday… I hope yours was more fun and less, uhh… interesting.

To-Go California, You can order your meal but you’ll never eat.

Consider this a bit of a PSA.

I stumbled across this article in The Wall Street Journal about some new regulation for fast food and dine-in restaurants no longer providing plastic utensils with your order, unless you specifically ask for them.

Gone are the days when you can blow through a Taco Bell drive through and be assured that you’ll be able to eat all your meal.

You’d think, “No worries, I’ll use my hands.”

TRUE you could, but then you’ll find that the napkins you assumed would be in the bag, AREN’T.

Not to worry though, It’s not like you’ll have hot sauce in the bag either because those too you’d have to ask for.

So, Taco Bell, and other fast food options will kinda be a no option without a checklist. I’ll attempt to help by providing my personal Ordering checklist. I’ve got it on my phone, but I’m thinking a post-it note stuck to my car dashboard might be better.

Fast Food ordering:

  1. Food
  2. Drink
  3. Condiments (Ketchup, Hot Sauce, etc.)
  4. Straw
  5. Napkins at least 5 (More depending on type of food)
  6. Necessary utensils (Spork, Knife, etc.)
  7. Pull up to window. Pay.
  8. CHECK that meal is correct and necessary Utensils, Napkins, and condiments are present.
  9. IGNORE HONKING of impatient people waiting behind you!

Yet again The State of California is working to make your life better, through unintended consequences.

Most of the time, If you’re a working stiff, perhaps hourly you’ve maybe got a 30 minute lunch. If the company you work for is exceptionally generous you might even have 45 minutes or a whole hour!

With traffic in most areas around California industrial parks, for a working stiff, it works out like this;

5 minutes to get out of the building, 5 minutes to get out of the parking lot (due to everyone else trying to leave for lunch) 10 minutes to navigate the rest of the lunch hour traffic from all the other companies in the industrial park.

5 minutes at traffic lights and turning into the nearest strip mall or gas station parking lot.

Pull into shortest line for for one of the fast food places, (Wendy’s, Mac Donalds, Taco Bell, Starbucks, Panda Express, etc) wait in line 5 to 10 minutes (By which time you’re already late if you’re on a 30 minute lunch.)

Place your order, 5 minute wait for food, then mad 10 minute dash, (You’re late at this point if you have a 45 minute lunch,) back to the industrial park.

5 minutes waiting at lights, 5 minutes to get into parking lot and find parking space, 5 minutes to get back into building. (You’re on the raggedy edge of being late at this point if you have an hour lunch.)

You get back to your desk, ready to resume work and eat your meal while you’re working…

You open your bag of cold soggy burger & fries or tacos that started out as crunchy but which are now, anything but. Voilà you discover that the whole exercise was pointless because even though you could eat without the condiments, or perhaps even the utensils, you have nothing to wipe your hands with.

Your lunch sits in the bag, not getting any fresher until finally it smells disgusting and ends up in the trash. Thereby contributing to food waste and spewing CO2 into the air, in the dash to get lunch that was also a pointless waste of energy.

So tell me again how wonderful it is that you’ve eliminated basic necessities to protect the planet? Huuuummm?

I swear to God, all of these jackass politicians should be under mandatory orders to live with their proposed laws for six months before they can put them into action.

Not ONE of the political elites in any California city has ever had to punch a clock or be screamed at by an overbearing manager over their lunch break being 2 minutes too long.

All one need do is look at the distribution of restaurants in and around industrial parks in San Diego, Irvine, Victorville, Huntington Beach, The San Fernando Valley, Ventura, or Los Angeles to see that most of the “working class” lives what I’ve described above daily. Or they bring their lunch so that they can at least have a real few minutes of rest during their lunch break.

Now, thanks to the politician brain trust, this new anti utensil or condiment law will ultimately slow the food ordering process down even further.

Not that these Politicians give a shit. After all, if the workers aren’t spending money on fast food, after skipping breakfast to get the kids off to school, it means the workers will have more money that can collected as taxes.

Obviously the low wage earner who collects $12 a week in unspent lunch money doesn’t need it now do they?

Plus, the Arch Ministers of Public Health can chalk up a “Win” because obesity will be less of a problem. Who cares if the workers are starving while they toil away to earn enough to pay their tax burden? That just means they’ll die sooner and a whole new group can be imported from wherever.

Maybe the Politicians are hoping it’ll be mostly white people!

I’m just not that into it…

There was a time in the recent past when I, like many other Apple users waited on pins & needles for the next iOS, iPhone, iPad, MacOS or MacBook computer.

This last iPhone event left me feeling sort of meh.

iOS 15 is somewhat interesting but I’m in no great hurry to update. This is very unlike previous years.

The iPhone 13 Pro is nice, but not dazzling. I sorta like the pale ice blue color but since I use a case that isn’t really all that compelling. My iPhone 12 Pro is still quite serviceable and does what I want it to do.

I’ve currently got an Apple Watch series 5.

I thought that I’d upgrade to the watch 7. I originally got an Apple Watch Series 3, then upgraded to the 5 and thought a 2 year upgrade cycle would probably be reasonable. Looking at the Series 7, I see no compelling reason to upgrade. Okay, it’s got a bigger screen and what else does it get me?

Apple’s presentation was even Ho Hum, it was almost like Apple themselves was “phoning it in“. Every Apple event they’re gushing about how impressed they are with their own technology. How each product, “…is the most advanced Technology Apple has created…”

Every time they say that now, I fight not to say, “Duh” out loud.

I’m always way more interested when they compare their technology to the competition.

I find myself wondering why I wasn’t excited, or even disappointed with Apple this year. I’d have expected to feel something one way or the other, I just don’t.

Early iOS adoption statistics might be indicating that I may not be alone. I caught an article last week, (which I can’t find now,) that said the one week adoption rate for iOS 15 was substantially lower than for iOS 14 in the same period last year.

Why have I not updated? That’s simple, I dealt with bugs that messed my phone up on both iOS 13 & iOS 14. I decided that this time around I’d wait. But when you add the whole issue of the CSAM scanning and privacy issues that honestly feel like an about face for Apple…

Well, I guess it feels a bit like betrayal.


It’s akin to that feeling you have finding out your spouse is sleeping with someone else. When you find something like that out, you tend to get angry, then when you calm down, you start evaluating the choices available to you.

Divorce is messy and ends up hurting you as much as your spouse. That’s kind of the nuclear option.

Ignoring it and hoping it doesn’t happen again, is another option. Sometimes that works, but can you ever really trust them again? Do you find yourself questioning everything they say or do for the rest of your life?

Couples counseling is another option. But that only works if both parties are willing to realize that a) they’re in trouble, b) the relationship still means something, c) are willing to give it a go and mend fences.

To some extent, Apple has behaved like the spouse that was caught screwing around. They’ve quietly admitted that CSAM requires more work. Their decision to hold off on rolling it out feels like, “I’m sorry I got caught. I promise I won’t do that again.

No spouse, ever totally believes that. The one saying it, knows that there will be an opportunity in the future that simply can’t be ignored. The one hearing it, knows this too.

(I’m specifically being gender neutral above, because I’ve seen women cheating on their husbands and men cheating on their wives. Interestingly, I’ve been the third wheel in both equations. Not really proud of that, it’s life, it just happens sometimes.)


I think right now, I’m personally at the “Ignoring it phase,” with Apple. I would consider the “Couples counseling,” phase. But I’m wondering if divorce from Apple is in the cards.

I have to wonder if this is the way a lot of Apple users are feeling. The article I mentioned, was trying to cover the lower adoption numbers by highlighting that unlike previous years Apple is going to be supporting iOS 14 and iOS 15 bug fixes concurrently. In the past when Apple released a new iOS they stopped development on the previous version.

This meant that there were no security patches or bug fixes for the older version and if you wanted to close those security holes, you had no choice but to upgrade the OS. This year is different. Apple has said they’ll continue the security patches indefinitely.

I’m sure that some of the upgrade hesitancy is due to this, but I seriously doubt that all of it can be explained away. This feels more like “coming home from somewhere you never should have been,” (Thank you Garth Brooks) and finding your pillow and some ratty blanket on the couch.

I’ll admit that pillow on the couch is way better than a confrontation in the driveway and a gun shot under cover of thunder in the distance. But it’s just as much a statement. (Yeah, I’ve gotten the pillow once or twice too.)

Right now, I’m giving Apple the pillow. We’ll have to see just how contrite they are, and how willing they are to keep my business going forward.

Gee Thanks Joe!

So as has been stated, I’ve been looking for a job for 2 years. I was laid off from my previous job due to outsourcing in Aug of 2019. Just in time for COVID Yea!

Throughout 2020 I applied for various jobs in my field that process has continued into 2021, with no positive results.

With over 30 years experience in technology and software testing I’m apparently unemployable. That’s not whining, it’s just a statement of fact.

Way back in the day, I demonstrated that I had a high aptitude for technology and computers. I was literally hired out of a junior college and never went back to finish a degree. That’s on me I admit it.

The fact is most of what was being taught in colleges had zero relevance to what I was actually doing because the colleges were teaching technology that was already 5 -10 years out of date and I consistently found myself on the bleeding edge of new technology.

There was little incentive or indeed value to paying for knowledge that was irrelevant. You see, my pragmatism dictates that knowledge is useful, a piece of paper proving my indebtedness, less so.

New hardware, new software, new languages, new technology was coming out every month. It was all most of us could to just to keep up within the companies we were working for.

While all this was going on, there were the innumerable layoffs, mergers, and acquisitions that made someone like me have a resume that looked like I couldn’t keep a job.

That assumption is completely false, but as time moved on and younger HR people came into the industry, they were applying all their college knowledge to my resume and frankly there was a significant disconnect.

Over time it got harder and harder to find a job, but I persevered and remained gainfully employed.

In my career, I have been a technician (back when we actually repaired machines at a board level), I’ve been a technical instructor (teaching others how machines worked and how to repair them), I’ve been a regional representative (supporting corporate product sales, and solving problems of product implementation and repair), I’ve been a technical support representative (explaining very technical issues to non-technical people to solve their problems), I’ve also worked in retail and warehousing.

All of this experience is useless today because I don’t have a degree. I ask you what function a computer science degree would serve since that degree would include FORTRAN-77 or COBOL, parallel communication, or RS-232? Virtually none of these skills have any relevance today.

To be sure, subsequent technologies that grew out of the aforementioned do have relevance and those technologies would have been learned on the job over the intervening years.

Which is to say, I’m on an equal footing with anybody coming out of college today with the possible exception of those educated in C# or some of the later languages. That being said, the core logic of computer languages is still the same. High level languages compile down to an instruction set that commands the processor to carry out specific actions. That hasn’t changed since computers occupied entire buildings.

The language simply provides a more human readable and therefore easier mechanism to create software. You can still bypass all of those higher languages and write software directly in assembly code. (The instruction set specific each processor. Before you ask, there are still people who earn really good money doing exactly that. Those folks, I respect immensely. Coding at that level is tedious and abstract beyond belief. I’m a little too ADD, or not ADD enough to do anything more than ‘tinker’ in that realm.)

As an aside, the folks that code at that level, are the folks ALL the sexy high level languages rely on. Assembly coders are an interesting and unique bunch. It’s not unusual for them to be socially challenged and challenging to more “normal” people. Assembly coders, deal in absolutes, mathematics, and the purity of silicon switching. Right and Wrong are terms which have no grey areas 2+2 in their world always equals 4. Something works and is therefore “right” or it doesn’t and is therefore “wrong”.

I haven’t met any coders who work in the new field of quantum computing. I would guess that they are a completely different kind of duck. Quantum theory being somewhat less determinant and the underlying math is so far beyond me I can’t begin to visualize it. I’m quite content to take it on faith that they know what they’re doing.

I digress…

The point is, I actually like computers and technology. I like testing it, and verifying that the code does what it’s supposed to do. It’s a puzzle to me. My job in the last decade or two has been to test the software coders create. I’ve mostly looked at this as a cooperative effort where I’m a fresh set of eyes working on code that the actual programmer finished working on weeks before. They’ve moved on to other parts of the project and are being productive. I’m making sure they didn’t miss anything and if they did, I’m the one who captures the errant behavior and shows them how I induced the error.

In this capacity, I’ve learned so much and been blessed to work with some truly amazing people. I earned the nickname “Demon” and wore it proudly. My “Demonic” tests, helped to produce a wide variety of award winning and useful products, of which I am also proud.

Now I’m dealing with having all that taken from me. It was hard enough to deal with human HR folks and get myself, or my resume in front of a hiring manager. Most hiring managers look at my resume and think, “This guy has been around. I’ll bet he knows all kinds of things that I could leverage.” That’s how I’d get hired.

But that was when humans actually read a resume. Now, folks like me are lost in the filters of HR databases. Databases I might add, that are controlled exclusively by the priesthood contained in the bureaucracy of layers of HR.

The hiring manager writes out a list of ideas about the kind of person they’d like to fill a position. HR passes the requirements up level after level of representatives ultimately getting the request to one or two people who actually enter the requirements into a database. However, by the time the data is actually presented to the hiring system the requirements are absolute and glacial.

The hiring manager wrote the initial requirements in a fuzzy way. “This or this would work for the position.” HR enters the requirements as “THIS AND THIS MUST BE PRESENT.”

(C) Scott Adams 2008

That’s how you end up with tons of people out of work, and corporate HR saying there’s no-one available for a particular requisition. This is also how you end up with corporations not capitalizing on workers within their own ranks for internal openings that could be promotions, and why so many workers leave a company after a few years.

I’ve personally witnessed newly minted Baccalaureates applying for open positions within the company they were already working for and being ignored. After 6 months or so they’re tired of being passed over and they go to work for the competition in the position their new degree qualifies them for.

Then I’ve also gotten nasty looks from HR when I laughed in their faces as they lamented that workers had no loyalty and how difficult it was to ramp new hires up to being productive. When asked what I find so funny, I’ve told them exactly why people in my department left. It wasn’t the pay, it wasn’t anything other than HR locking them into a particular position and having no hope of advancement.

As a side note HR people really hate having their noses rubbed in their own poo.

All of this is what workers deal with every day. This is one of the reasons the hiring process is such a royal pain in the ass, for employers and prospective employees.

But along comes Joe Biden… With his imperial decrees about mandated vaccines, and what do we have?

Now we have companies adding “Must Be Vaccinated” to their job openings.

And in HR’s usual moronic fashion, they’re making the vaccine mandate apply even to people who are applying for remote only positions.

So now there’s another hurdle to surmount. It’s not enough that the remote position is on the other side of the country and a prospective employee will never darken their corporate door. A remote only worker cannot transfer any disease to any other employees via a zoom teleconference. There is zero risk or threat of contamination.

Yet, corporations will demand that a worker be vaccinated simply because they can. Their HR departments will fall back on the excuse, “We’re in compliance with The President’s mandate, and the law. After all it’s for your protection…”

Translated: “We’re protecting the corporation from any/all liability. We’re doing our bureaucratic duty…”

So thanks President Biden, you’re batting 1000 at screwing everyone. Good job!

(AP Photo/Charlie Neibergall)

The trouble with career politicians is that they’ve never had to consider unintended consequences. That fact coupled with greed and elitism is why politicians always fail.

If good ol Joe keeps up the pace of failure, I suspect that Washington DC will very soon look like The Vatican. The question will be, are the walls keeping the angry population out, or the shitty politicians in?

Apple got the message

At least for now.

Over the weekend, I noticed an article reporting Apple had decided to delay their roll out of CSAM scanning.

The report dropped on Friday and it’s unclear if Apple will push forward at a later date but for the moment Apple has put it on hold.

I think it’s a good thing.

It’s also unprecedented.

Apple never backs away from something they’re planning to release. At this point they’re saying that they’ll invite opinions from privacy protection groups and the “feel” is they’re going to be inviting input from developers in the privacy arena.

This doesn’t mean that they’ won’t push forward at a later date, but it does mean that for now, our devices won’t be spying on us, (any more than they do already,) I’m calling it a win.

That being said, I’m going to be watching privacy settings very closely and being a lot more alert to any changes.

I don’t have anything to hide but at the same time I’m not willing to just leave my digital doors and windows open all the time either.

After all, who doesn’t from time to time get a naked photo of a friend? LOL!