Failure Testing of Complex Distributed Systems: Confirmable Experience?

The article, “Abstracting the Geniuses Away from Failure Testing” puzzles me.

There appears to be agreement that the inability to demonstrate the durability and graceful degradation of distributed protocols is a matter of failed abstractions.  At the same time there is consternation over the inability to have perfect failure detectors because “it is impossible to distinguish between delay and failure” in distributed systems.  There is more explanation of this situation in the article.

A key issue is the difficulty of ascertaining that compositions of fault-tolerant components results in distributed fault-tolerance, and how is that to be confirmed (or demonstrated to fail).

Enter Fault Injection

The idea of fault injection appears to be the practice of intentionally perturbing a system to see how it responds.   My brain is apparently too small to cotton to this notion.  In any case, the limitation is that it does take brains to conceive of promising perturbations.   There is also the matter of this being a form of black-box testing, working from the outside of the system and attempting to isolate cases where assured behavior is violated.

It turns out that it takes considerable ingenuity and labor intensity to come up with revealing fault injections.  

To have a chance at automation of this kind of testing, advances in instrumentation are supposed.  The money quote seems to be this: “A system is fault tolerant if it provides sufficient mechanisms to achieve its successful outcomes despite the given class of faults.”

At this point, it is not clear to me how this doesn’t revert to the previously unsolved problem with regard to establishing success and classes of faults.  The limitation is mine, I’m thinking.  There is promising success involving a kind of automated big data analysis of detailed operation of the system’s operation and use of fault-injection infrastructure.  System redundancy seems to be a factor in the case of large-scale distributed systems.

I would prefer to leave such large-scale considerations to others. 

Small is Not So Beautiful Either?

I do wonder whether I need to comprehend this with regard to scaling up of a tiny system to a grid of distributed ones.

Right now, the critical-for-me problem concerns a tiny non-distributed system.   I need to confirm that a specified API and the behavior behind it is satisfied by an implementation.  This is the classic testing case of not being able to demonstrate correctness of the implementation, but possibly demonstrating a failure.  The problem is to figure out what tests demonstrate a kind of cover of correct behavior, and where can one probe for likely bad outcomes that would reveal a mistake.

This is a white-box problem.  I have the code of the implementation and I have the specification.  If there is a “bug,” it is likely in the specification because it is relatively easy to inspect for its faithful embodiment in code.  It is a very small system, and kept small, and rigorously-specified, for that reason.

For a defect to be apparent, it seems that I must have higher-order requirements to satisfy with respect to successful outcomes and “classes of faults.”  Here there is food for thought, although not because I have a distributed case to deal with.  Yet.

I am left to ponder whether there is something about the scaled-up distribution cases that I must anticipate fairly early in the miniature cases I work on first.

Posted in Computers and Internet, Golden Geek, Miser Project, nfoWorks, trust | Tagged , , | Leave a comment

Returning to the Moon: Static Test

After restarting with a fresh setup of the wingnut/ blog authoring folder, the first modifications are enough to have the blog appear with the correct identification and configured for the destination web site.

Just-Enough Initial Configuration

The next step is modifying the default wingnut/ authoring setup and its Hexo landscape theme by specifying the destination blog and its web location.  Changes are to the initial wingnut/_config.yml (YAML) file installed by Hexo. 

File wingnut/_config.yml is annotated with comments and then customized to establish the title, author, language, timezone, and intended web-site location.

The file is also annotated with comments.  Version-numbering is applied to identify changed lines and when they were introduced.  This file is under source-code management in accordance with my Visual Source Safe Adjustments

The changes establish the blog title, author, language, and the time zone for date-time stamps on posts.  The location of the blog at its hosted web-site is also specified so that cross-references and access to components is accomplished.  Nothing more is needed to have the blog appear correctly alongside the legacy blog at

Generating and Publishing Just-Enough


After confirming the adjustments using local “hexo server” a “hexo clean” operation precedes “hexo generate”, resulting in a fresh wingnut/public/ folder for the new configuration and destination.

Hello, Spanner Wingnut

After FTP transfer of the new blog material, access is confirmed over the web.

The hosted-site wingnut blog folder correctly identifies the blog and serves the "Hello World" post supplied as part of Hexo initialization.

We are not ready to make this the default blog at that location.   The next steps include progressive customization of other features of the Hexo setup and of the initial theme.  The possibility of switching to a different theme remains open.

Bonus Points

The browser of my smartphone sees an appropriately-adjusted presentation of the blog page.One feature of the Hexo static generation and the available themes is having responsive blog sites.  Curious to see how that works “out-of-the-box,” I accessed the Hexo-generated blog home page using my smartphone.  Voila!

The responsiveness is impressive enough to tempt me to tip the smartphone on its side. 

These results are wonderfully automatic.

(Differences in the tonalities are the responsibility of the photographer’s attempt to soften the contrast of the original.)

Turning the phone horizontally automatically adjusts the blog page to the new orientation and line width, automatically fitting the page content in a pleasing flow.

Posted in Blog Development, blogs, Golden Geek, Spanner Wingnut, Toolcraft | Tagged , , | Leave a comment

Returning to the Moon: Vertical Assembly Scaffolding

Having constructed and used authoring folders for Hexo a few times now, I see that it would be good to start with a clean slate.  I will work forward from the default setup until I have a customized blog theme that satisfies my needs.

Once Again, with Feeling

Purging my wingnut/ authoring folder and also destroying the corresponding managed code in my VSS repository, I start once again.

This time, I am going to follow the Hexo quick-start procedure to the letter.

The 5 comments to initialize an initial authoring folder with Hexo and serve it locally for browser viewing

Latest Node.js, Fresh Hexo-cli

There were security updates this week.  I took that opportunity to upgraded to Node.js 8.8.0.  I then installed hex-cli 1.0.3 to complete the first step.

Whether hexo-cli might already be installed, the installation is performed again after upgrading to node.js 8.8.0 and emptying the wingnut folder.

It is also satisfying that NPM on 8.8.0 is not so verbose, only reporting what I might need to know.

Initializing wingnut/ Anew

I’m already working in the wingnut/ folder.  Now to request initialization of it as a Hexo folder for authoring and blog generation.

The Hexo init operation is requested with the wingnut/ current folder.  The folder will be initialized with a default theme and the Hexo authoring structure and Hexo files

I’m not unclear what the npm message about committing package-lock.json is all about.  I will import it into the Visual SourceSafe backups.  I assume that leaving the file as read-0nly will not create any problems.   If I’m mistaken, that is easy to remedy.

Once More to the Brink, NPM

At this point, the Hexo quick-start procedure specifies performance of an “npm install” operation.  It seemed pointless to me and I have skipped this action in the past.  Today, I am going to follow instructions and see whether it matters.

Following Hexo initialization of a blog-authoring folder, an NPM install is performed from within the folder.  It matters.

Something does seem to have happened. 

Now perform “hexo server” and then go see what the blog looks like here on my local PC.  Without any configuration work, it is not set for publishing to the site, but it can be viewed here.

All-Assembly Mockup Test

There it is.

The pre-initialized blog and single "Hello World" page presents a greeting and tips for further activities.  For wingnut, further configuration is needed before the blog can be published as the author's own.

… to be continued

Posted in Blog Development, blogs, Golden Geek, Spanner Wingnut, Toolcraft | Tagged , , , | Leave a comment

Returning to the Moon: Launch-Pad Repaving

[update 2017-10-25T14:43Z with touch-up of preamble]

I’ve demonstrated two important end-to-end capabilities for migration to static generated blogs.

  • A Hexo authoring folder can be customized for replacement blogs that, when generated, can be uploaded to my webserver along-side the legacy blog content that I want preserved.
  • Visual SourceSafe 2005 is usable to backup and source-code manage my customizations and content material for the generated blogs.

Recall that much of this precautionary effort is to deal with situations when there are potential collisions via updates to the authoring folder via NPM and also to themes via updates of clones from GitHub.  Along with that, there is source-code management of material that I create or modify, as well as backup of the generated blog folder.

Reset and Learning Curve

I decided to clear my wingnut Hexo authoring folder on my PC and rebuild it, starting with the pure Hexo installed default configuration with its initial theme, landscape.  I will use its examples and my own customization efforts to comprehend three aspects of Hexo-generated blogs.

  • Creating .ejs (Embedded JavaScript) files that generate as theme-customized HTML pages, experimenting with those employed in the default Hexo theme files.
  • Using.styl (Stylus stylesheet language) files to generate custom CSS, starting with experimentation with those employed in the default Hexo theme.
  • Installation of plug-ins, their enabling and their configuration in the Hexo and theme _config.yml files.

I won’t be using any alternative layout and template systems if I can help it.

Visual Source Safe Adjustments

As part of restarting the wingnut/ authoring folder with an initial Hexo-originated configuration, I was led to adjust preservation/backup/source-control with Visual Source Safe. 

  • There are binary file formats that VSS needs to be told about.  I added them to the SourceSafe Options File Types for Binary files:
  • Visual Source Safe produces hidden and read-only vssver2.scc files in folders that it touches.  There are two difficulties with these.
    • In folders that have content transferred to the generated public/ folder, any vssver2.scc files are transferred by Hexo.
    • Some Hexo operations attempt to consume the vssver2.scc files, aborting because the format is not understood.  This happens even though the files are hidden.
    • The workaround is to clean out the *.scc files after using Visual Source Safe on the Hexo authoring folder.  That can be accomplished with a single console command.  It will be built into a .bat script that I will add to the authoring file for maintenance purposes.  The command is simply “del /S /ARH *.scc”.
  • Many of the files in the authoring folder will not be touched by me.   I am segregating my use of Visual Source Safe to accommodate three cases:
    1. Files, such as configuration files and other customizable files and folders that only I will modify are checked into VSS and will be checked out for author modification.
    2. Files and folders that only Hexo maintains for internal purposes will not be checked into VSS. 
    3. Files and folders that Hexo produces for author usage, including public/, will be checked into VSS.  But they will be left in checked-out status after every check-in.  This allows the material to be backed up although immediately modifiable by Hexo and/or the author.
  • When updates are available, there may be transition arrangements that require cases (1-2, above)  to be adjusted so that (1) updates are not blocked and (2) pre-update content is checked into VSS as a precautionary backup.
Posted in Blog Development, blogs, Golden Geek, Toolcraft | Tagged , , | Leave a comment

Returning to the Moon: Launch-Failure Safety

Proof-of-concept for deployment of Hexo-generated static blog folders on my Windows 10 PC to one of my hosted-site blog folders is confirmed.  I can set up an optimized 1-2-button publish mechanism.

The next proof-of-concept is to have a source-code management and recovery mechanism for Hexo-operated authoring folders. 

I’ve confirmed that Visual SourceSafe 2005 is usable on the Windows 10 PC where I will perform authoring.    The VSS repository is located on a separate local machine that houses my development web server where there are recovery-usable mirrors of my public, hosted web sites.

Backup and source-code version control using the web-development VSS repository covers all but one folder of the experimental authoring folder.

The VSS client on Windows 10 shows the HexoWingnut folder holding backups of the wingnut/ authoring folder now under source-code management

The hexo-created wingnut/node_modules/ folder cannot be backed-up to the $/HexoWingnut VSS Project folder.   VSS prohibits file names starting with “$” and there are some of those.  Fortunately, there is no need my source-code management of node_modules.  I won’t be touching it.  Recovery can be from normal backups if not Hexo.

Although Visual SourceSafe is no longer under development and there are more appealing source-code management approaches these days, it is very handy in this case.

  • I already have it and I have workflows where it is already tied into support of my web-site development and mirroring of the material published to my hosted web sites.
  • Using VSS source-code management for my work in the authoring folder, including modifications of the Hexo and theme materials, will avoid collisions with parts of the authoring folder that are cloned from GitHub and may be updated from their GitHub origin.

Next Steps: Customization of a Hexo theme for satisfactory blog operation, writing about my current projects,  and removal of Movable Type software from my hosted web site. 

Posted in Blog Development, blogs, Golden Geek, Spanner Wingnut, Toolcraft | Tagged , , | Leave a comment

Returning to the Moon: Mercury Retro

For restoring my blogging capability using modern static-blog generation, I have completed an initial end-to-end proof-of-concept: confirmation of direct-publishing capability from my Windows 10 PC’s authoring folder to my hosted-site blog-folder location.

End Points

  • The launch pad is a Hexo-created authoring folder, wingnut/, on my Windows 10 PC.
    • There is a barely-usable initial configuration of theme Polar Bear.
    • The Hexo-provided “Hello World” post is all there is.
    • Operation “hexo generate” produced blog-site content with that single post in authoring-folder location wingnut/public/.
  • Splashdown is into my hosted-site location of blog “Spanner Wingnut’s Muddleware Lab” (“Wingnut” for short).
    • “Wingnut” is a sandbox for experimentation and confirmation of blog setups before I migrate more-serious blogs.
    • It preserves the original, Blogger-produced “Spanner Wingnut” posts, archives and feed.
    • It preserves the subsequent sandbox work using Movable Type posts, archives, and feed.
    • The deployment of Hexo-generated material to this site location shall not splash on any of that material.
    • This situation applies to all locations where I have existing blogs to migrate.

Sub-Orbital Transfer

The Windows 10 Pro File Explorer provides file-folder access to FTP servers.  Simply type the ftp name (here, in the location field and click Enter.

FTP login is by typing the ftp server location in the File Explorer location field and supplying login credentials

The dialog has not been modernized along with the evolution of Windows.  It appears to be a still-supported legacy feature.  It will do the job.  The warning about FTP making unencrypted transfers is also important to appreciate.

After log-on and navigation to the proper folder, files can be copied manually.  That’s sufficient for proof-of-concept.  Refinement comes later.

The five folders and two files of the generated author material (first folder view) on the PC are dragged to the splashdown folder reached by FTP (second folder view below).

The authoring folder generated public/ content is selected and dragged to the FTP-accessed hosting-site location without any conflicts, leaving current material intact

None of the Hexo-generated material splashes over existing material of the splashdown location. 

Safe Landing

The Hexo-generated blog’s entry page is at index.html.   The safe landing has the expected minimal page at first.

The Hexo-generated home page displays properly from the uploaded index.html location

Following the “Hello World” link successfully reaches the full content of the test post.

The Hexo-generated initial post is correctly linked from the  blog home page, showing the desired light content of the Polar Bear theme

Although the blog page “Archives” and “About” links are not connected (and no About page has been authored), the Hexo-generated archives folder is present and occupied with a correct entry and link.

The Hexo-generated archives folder is not connected to the other pages.  It is present and connects to the default initial page supplied by Hexo.

I am declaring victory for proof-of-concept.  Streamlining and optimization of the upload process comes later, along with further customization and organization of the blog.


  • Hexo-generated blogs uses index.html for the blog home page.  
    • That is necessary for local “hexo server” viewing of the authoring folder.
    • On the hosted-site, I had intentionally made index.html subordinate to index.htm which is subordinated to default.htm in the search rule for folder default pages.  Automatic production of indexes at the server is not permitted.
    • Requiring index.html  is a fortunate coincidence.  It allows for a sandbox for static-generation confirmation without interfering with existing content.
  • My site architecture requires default.htm for the blog home page.  The index.htm page is reserved as part of site-wide scaffolding and construction annotation.
    • Before migration to Hexo-generated static material, I will preserve the existing default.htm page under a new name.
    • The production deployment scripts for Hexo-generated site content will  copy  public/index.html to public/default.htm before posting to the hosted site, blending into the site-wide architectures.
    • Migrated Hexo-generated blogs will provide links to the archives, feed, and last post produced earlier with  Movable Type and/or Blogger.

Onward to the next challenge.

Posted in Blog Development, blogs, Golden Geek, Spanner Wingnut, Toolcraft | Tagged , | 1 Comment

So Close, Yet an FTP Too Far?

I am continuing my investigation of Hexo, a static blog generator that has these useful characteristics:

  • Hexo installs and runs under Node.js and runs in JavaScript.
  • The generated public/ directory consists of static material that can be uploaded and served from any conventional web server, with GitHub a popular choice
  • There is an ecosystem of plug-ins and customizable themes

There are some worrisome concerns around how to customize the blog.

It is important to stop fretting about perfecting the blog though.  Instead, I need to confirm an end-to-end approach that takes a proof-of-concept simple blog through the workflow necessary for publishing on my hosted web sites.  Only then should blog design be addressed.

Here’s how I am explaining all that to myself.

Authoring Configuration

  • A local authoring folder is created on my PC.  The authoring folder is set up by Hexo for generating web pages of a single personalized, customized blog.     I will have an authoring folder for each of my published blogs, along with some number for experimental purposes.
  • A fresh Hexo-initialized authoring folder includes a pre-installed default setup for generation of a simple blog that can be viewed locally and authored immediately.
    • Verify installation and first-run by viewing the pre-installed “hello-world” initial post via the default setup.   
    • An initial theme,  “landscape”, is pre-installed and enabled.  This establishes the appearance of the generated web pages and other aspects of the published blog site.
    • The initial configuration will need some customization for the title and published location of the generated blog. 
  • Practically everything in the authoring folder can be customized.  
    • Some customizations must be enabled by NPM installation of plug-ins for Hexo to employ as directed by edits to configuration files.
    • Installation of theme definitions in the authoring folder is typically by cloning them from GitHub projects.
    • Themes also require configuration and may be customized further.
  • It is unclear how to manage and backup an authoring folder.
    • Providing preservation and source-code management of local customizations and of the authored sources of blog content.
    • Providing for possible updates to downloaded themes that have been customized locally.
    • Providing for possible updates to the Hexo-provided basic configuration and its initial content that may have been customized locally.

Not So Fast, Sparky

I have several requirements for authoring new posts of existing blogs of mine.

  • Atom feeds of each Hexo-generated blog, with the feed locations already used
  • Root-entry of the blog provide at least the latest full post on the default page, with links to the chain of previous posts and the archive of posts by year and month
  • Previous – Top – Next post links for navigation are valuable
  • Having Tags and Categories
  • Having DISQUS for comments
  • Having a light appearance with legible colorization of code blocks and pleasant quotation blocks
  • Providing social-network information, an author page, and similar information

All of these are achievable with existing Hexo plug-ins and themes.  Having them all in combination with an authored appearance, a private theme, that works for me is the challenge.

I have been dithering around learning how to customize an authoring configuration that satisfies all of my requirements.  What I am learning is how little I have of the tacit knowledge that the authors of templates must have.  I also don’t know where there is  documentation that I need.

I am also fretting over the wrong problem. 

What I need most of all is a nearly-one-click mechanism where I can author a blog post, have it generated, and have it appear near-instantly on my hosted site.  That’s what I have right now as I use Open Live Writer to compose and then publish this post, even though it goes to and not any property of mine.

With the point of generation being on my local PC, the question is, “How do I get deployment from there to the hosted site with 1-2 clicks?  How do I integrate that into my site mirroring and backup arrangements?

I need to stop fussing about having the generated blog be perfect and start with something that is minimally adequate.  There is no point in further refinement until an effortless deployment arrangement has been confirmed.   That’s the critical proof-of-concept.

Deployment Concerns and Approach

  • I want to publish in the same location as my existing blogs, in a manner that preserves the previous content
    • So far, there is no conflict with the material that Hexo generates, in the authoring folder public/ subdirectory, so the Hexo-generated blog content can sit side-by-side with the older pages and archives of the previous version. 
    • Hexo uses index.html as the default root page of the generated blog.  For technical reasons, I may have to use default.htm.   I can work around that on my servers without changing anything in Hexo.
    • I do need a publishing mechanism that will leave the old-blog material intact when uploading new/updated Hexo-generated blog content.
    • I need a backup and source-code management system that covers a complete authoring folder.
  • Direct Publishing
    • I will directly FTP new generated public/ files to the hosted site.   There are a number of ways to do this.  I will want to find one that  can be turned into one-click operation via a batch script or by use of the hexo deploy command.  
    • Having an effective direct-publishing approach needs to be confirmed first.
  • Backup and Source-Code Management
    • If the Visual Source Safe repository co-located on my web development server can be reached from Windows 10 on my PC, I can put entire authoring folders under source-code management in appropriate Mirror VSS projects of the nfoCentrale Web Deployment Pillars.  
    • This protects against customization loss when Hexo themes installed by cloning from GitHub are updated from GitHub. 
    • This, with appropriate included documentation, also supports any need to completely re-initialize an authoring folder.
    • The use of VSS in the Hosted-Site Mirror Projects also supports restoration of the blog (and more) from a mirror of the hosted site.
    • This or an equivalent is needed before I risk too much authoring-folder customization without any good backup, recovery, and source-code management.
    • Having an effective source-code management and recovery approach needs to be confirmed next.

Simple early steps.  Fail early if fail I must.  Recover quickly.

Posted in Blog Development, Toolcraft | Tagged , , , | Leave a comment