Menu

Michael Gracie

A beginners’ guide to running blockchain full nodes using external storage devices

For those loath to run an established (read: huge) blockchain full node because they don’t want to reserve double digit percentages of their laptop’s soldered (additional read: not upgradeable) solid state storage for the cause, here are simple steps to do so by leveraging external drives. These instructions are for macOS, but Linux and Windows users should be able to derive their own setup from it.

First, let us note the location and size of comprehensive blockchain data stores as presented by both Bitcoin Core and Ethereum Wallet

Bitcoin Core default stores the Bitcoin blockchain under ~/Library/Bitcoin. There you will find several directories and files, including those for your wallet. That which we will want to deploy elsewhere is as follows:

1) ~/Library/Bitcoin/blocks at roughly 170 GB

2) ~/Library/Bitcoin/chainstate at roughly 3 GB

Ethereum is less unwieldly, but nonetheless a lot of data. It can be found under ~/Library/Ethereum; we’ll concentrate on the directories below:

1) ~/Library/Ethereum/geth/chaindata at roughly 80 GB

2) ~/Library/Ethereum/geth/lightchaindata at roughly 0.5 GB

Note that we are going to ignore other items under the main directories, including that representing keys i.e. wallets, wallet.dat in the case of Bitcoin and the /keystore for Ethereum. Why? While setups vary, my particular environment includes a laptop with FileVault2 activated, and a discrete backup drive encrypted AES-256 that receives a weekly clone of the former. Hence I always have two very secure copies of my keys and prefer to keep it that way.

Post-Identification

We are now going to move the big directories noted above to an external storage device. To keep everything organized, first create a “Bitcoin” and/or “Ethereum” directory on the device. Then copy the entire directories from above into those. Next, append “.old” to the original directory location names i.e the directories you copied from.

Now we will create symbolic links aka aliases to point the wallet applications to the new data store. Type the following into terminal …

For Bitcoin

ln -s /Volumes/XYZDRIVE/Bitcoin/blocks ~/Library/Application\ Support/Bitcoin/blocks

ln -s /Volumes/XYZDRIVE/Bitcoin/chainstate ~/Library/Application\ Support/Bitcoin/chainstate

For Ethereum

ln -s /Volumes/XYZDRIVE/Ethereum/chaindata \ ~/Library/Ethereum/geth/chaindata

ln -s /Volumes/XYZDRIVE/Ethereum/lightchaindata \ ~/Library/Ethereum/geth/lightchaindata

(where XYZDRIVE is your drive name)

Note that as the latter locations go for each blockchain, they are small in comparison to the others (not critical to move). Further, with the Ethereum light client data (still in beta), it may be advantageous to not move the lightchaindata directory, in case you are on the road and want to use your wallet in a jiffy.

Finale

Run Bitcoin Core and/or Ethereum Wallet. Assuming your previously stored blockchain(s) is not too far behind with syncing, it should start rolling right away. You can delete the “dot old” directories after you’ve confirmed all is right with the world.

MG signing off (to contribute to the stability of another network, without the storage headaches)

Grab and validate National Vulnerabilities Database updates

Here is a concoction to grab National Vulnerability Database feeds, specifically the Modified JSON and related metadata, then validate the reported sha256 hashes:

import urllib.request
import gzip
import hashlib

#json file
fileurl = 'https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.json.gz'
json_file = '/Users/laptopuser/Documents/Active/NVD/nvd-data/0326/nvdcve-1.0-modified.json.gz'
urllib.request.urlretrieve(fileurl, json_file)
json_file_open = gzip.open(json_file, 'rb')

#meta file
fileurl = 'https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.meta'
json_meta_file = '/Users/laptopuser/Documents/Active/NVD/nvd-data/0326/nvdcve-1.0-modified.meta'
urllib.request.urlretrieve(fileurl, json_meta_file)
json_meta_file_open = open(json_meta_file, 'r')

#get hash from meta file
for line in json_meta_file_open:
    li = line.split(':')
    if li[0] == 'sha256':
        ze_sha = li[1].strip('\n')
        print('Meta:', ze_sha)

#calc hash of file
sha256_hash = hashlib.sha256()
with json_file_open as f:
    for byte_block in iter(lambda: f.read(4096),b""):
        sha256_hash.update(byte_block)
    ze_hash = sha256_hash.hexdigest().upper()
    print('Calc:', ze_hash)
    
if ze_sha == ze_hash:
    print('MATCH')

json_file_open.close()
json_meta_file_open.close()

You will get output that looks something like this …

Meta: E3ECE7D603F091E68E60E68CD6E230A28BC9E23EFB7E9B8145E559D1910BE9A6
Calc: E3ECE7D603F091E68E60E68CD6E230A28BC9E23EFB7E9B8145E559D1910BE9A6
MATCH

No apologies for the basic code presentation, nor for using urllib.request.urlretrieve. Feel free to copy and paste into Jupyter notebook or PyCharm if syntax highlighting is desired; as the latter goes, I know that function is supposed to disappear but my application requires keeping a sizable rotation of NIST’s handiwork close by.

MG signing off (to grab and validate some more)

Plugging mcrypt into PHP on macOS High Sierra 10.13: A Requiem

Since Apple’s desktop operating system was called OS X Leopard, I’ve been concocting little ditties on how to plug mcrypt functionality into PHP without complete recompilation and/or using a black box MAMP offering. The previous stop on that [eight year?] journey was with macOS Sierra. And it was/is the end of the line.

Queue list of excuses …

  • I don’t use PHPMyAdmin or Magento eCommerce (both major targets as PHP + mcrypt needs go)
  • I’ve been leaning on Linux VMs for relevant development work (trashing environments and starting over less often than ever)
  • I’ve taken a liking to Python, as it covers everything from number tumbling and data crunching to stupid little utilities for replacing website photos (dropping Anaconda in is a breeze)
  • And not really last but certainly not least, Apple’s latest release, High Sierra 10.13, came complete with a security-related flaw so ludicrous that I probably won’t upgrade until at least iteration two or three. It’s a trust thing.

For those who have already gained elevation in the States’ westernmost major range, all is not lost. If you run through the last set of instructions and replace any code that says “10.12” with “10.13” – along with identifying the proper XCode and PHP versions at the start – you should be ok.

Meanwhile, thanks for the good times.

MG signing off (because that’s all folks)

UPDATE 11/28/17: Like I said, lack of trust.

Right now in Houston

According to the Houston Food Bank

“Right now, one in five people in southeast Texas don’t have enough to eat.”

Probably more like one in three right now, so please click the logo below and give what you can. Link is via convio.net, and has been tested (i.e. I received a valid receipt directly from the HFB).

Medical supplies are also needed, and eBay/Paypal are matching donations up to a quarter million. You can contribute RIGHT HERE.

MG signing off (thinking Texans are pretty cool)

UPDATE: And here’s a Paypal link for Irma related relief -> https://www.paypal.com/us/fundraiser/107725536494624790

Programming the Yaesu VX-8DR on macOS, with the help of CHIRP and Valley Enterprises

In a word, easy.

The quad-band Yaesu VX-8DR is a nifty little radio, but in many respects it also suffers from feature overload. Hence menu pain. Even after several sessions of RTFM, one can still be lost. Programming via computer, please enter stage left on queue.

Unfortunately for Apple users, most pre-packaged programming utilities i.e. cable and software offerings cater to the Windows operating system and/or are designed for serial ports (think DE-9 connector). Those jokers with a spare MacBook Air lying around are almost SOL; they have to assemble their own tools instead.

With a little research it’s a moderately low effort endeavor. Even less if you just continue on here.

(more…)

Bring Blogger images into WordPress, the hard way

You migrate from Blogger to self-hosted WordPress. Your posts move over just fine, but for some reason (or another) your images forget their bus pass. Those pornographic stupid cat, hastily-prepared food, and trying-to-make-people-think-you-are-wealthy-instead-of-deep-in-debt vacation photos still show on the new site as they are properly referenced in the posts, but they actually remain on Google’s servers. You (or your client) don’t like that.

Meanwhile, the two plugins you found to solve this problem, Archive Remote Images and Cache Images, haven’t been updated in years. You take your chances anyway because you are lazy (if it is a personal site), or consistently over-promise and under-deliver (due to the impossibility of getting real work done at coffee shops). Either way, you must now hope you made a full site and database backup beforehand. If you did, you’re solution is now staring you in the face.

The script I concocted (shown after the jump) will get you a folder full of those images – with clean and pretty naming conventions – that you can upload to your wp-content directory, along with a SQL script to update links in your WordPress posts. Said programmatic wizardry dirty hack is written in Python – debugged using version 3.5.2 Anaconda custom (x86_64) on macOS 10.12.3 to be precise – and does rely on some SQL prep work. If you do not know Python, SQL and how to navigate directories while a terminal prompt blinks back, you have two choices: Google it (after determining what the definition of “it” is), or inquire about retaining me to do your work for you.

I’ll make the decision whether to continue easy too; if you cannot execute the following block of code sans assistance you are officially deemed “without paddle” …

SELECT * FROM `wp_posts` WHERE `post_content` LIKE "%blogspot%"
INTO OUTFILE '/home/dump/blogspotposts.csv'
FIELDS TERMINATED BY '|'

That look easy? Then proceed.

First, decide whether to run on your desktop (for future upload) or directly on server. Next, create a directory underneath where the script is located called /bspics. Lastly, make sure the directory the code is in is writable by all.

The code can be found here -> processblogspotimagelinks.py

Once you have changed the obvious stuff to suit your need, run it. Your /bspics directory will fill up with those images I promised – you can then place that entire directory underneath /wp-content – and you’ll also have a file called bsreplacescript.sql which you will run against your WordPress database to update image links in the associated posts.

Important [final] note: the coding was an iterative process, and some data analysis was done between steps in order to account for string possibilities encountered, generating clean file names, etc. It could be refactored, but wasn’t because 1) the end result works as intended and 2) removing those iterations would handicap attempts to modify it for a different data set.

MG signing off (to solve some not-so-commonplace problems)

“Comply with all crew member instructions” and then some

FACTS: People all over will travel all over this holiday season. They will stuff themselves with holiday joy. They will feign appreciation for useless gifts. They will complain about service, especially that purveyed by airlines.

This is but a public service message reminding everyone to comply with all crew member instructions, as well as those printed on the wing of the plane …

do not step out of this area

DO NOT STEP OUT OF THIS AREA (Photo: Yours Truly)

MG signing off (’cause that next one is a doosie)

Plugging mcrypt into PHP on macOS Sierra 10.12

We climbed up then down El Capitan, yet little changed. We’re still in the Sierras, and there’s still work to do.

The following instructions are for peeps a) developing on macOS Sierra 10.12, b) need mcrypt for their PHP development i.e. PHPMyAdmin and/or encrypting personally identifiable data before storage (unlike your health insurer, Yahoo! and/or pretty much every other knucklehead out there), and c) do not want to recompile PHP or run MAMP. Mcrypt will load dynamically with PHP after completing this tutorial.

We begin by acquiring the following stuffs …

1) libmcrypt-2.5.8, which you can find here; download by clicking the one of the two file links (author used “libmcrypt-2.5.8.tar.gz”);

2) PHP 5.6.24 source code, which is available here; [NOTE: you may someday update OS X beyond 10.12, and PHP may get updated along with it; the author used 10.12, and PHP 5.6.24 was included with that OS version. If necessary use php -v to check your version of PHP and then download the PHP source for that version.];

3) Xcode 8.0, which you can get from the App Store. You will also need the Command Line Tools (macOS 10.12) for Xcode, which you get by selecting “Xcode/Open Developer Tool/More Developer Tools…” from the Xcode menu, then logging into your Apple Developer account (yea you need one of those too); and

4) Homebrew (http://brew.sh) which can be installed by typing ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/ install/master/install)" at the command line.

But before we really wind up, let’s turn off System Integrity Protection (SIP):

  1. Click the Apple menu (upper left hand corner of your screen)
  2. Select Restart, then hold down the Command-R keys to boot into Recovery Mode
  3. Select the Utilities menu and then Terminal
  4. In Terminal, type csrutil disable and then hit return; then close Terminal
  5. Click the Apple menu and select Restart

NOTE: When you are done installing mcrypt, you can restart SIP by following the above steps while substituting csrutil enable in step 4.

Proceed.

(more…)

Debt crazy. Make that insanity.

“Insanity: doing the same thing over and over again and expecting different results.” – Albert Einstein

Grant Williams does a heck of a job explaining the situation

MG signing off (because owing nothing means you can)

Blocking Facebook, Completely

A long while back I deleted my Facebook account. Yes it was a large pain in the ass, but not nearly as bothersome as the potential risk of having such a thoroughly untrustworthy group – downright devious if you consider they change their privacy terms seemingly by the hour – holding a constantly updated dossier of oneself. I have avoided the site like the plague since. If I accidentally passed by I would immediately clear browsing history and caches.

Nevertheless, a few months back I observed browser cookies showing up from Facebook, so I added a Little Snitch rule to block all processes for facebook.com.

fb-ls-rule

Soon after I noticed that Instagram photos embedded in sites no longer displayed in my browser. Bonus! I was henceforth free from drunken selfies, cat gonads, and $30 spinach salads laden with moldy cheese and anchovies.

But the question lingered: How many other points of entry might not be blocked here? After all it’s a huge ‘organization’, and they’ve probably conjured a myriad of stealthy domains for the purpose of pinging, tracking, recording, and otherwise intruding. Thankfully I wasn’t the only one asking.

Enter one Jonathan Dugan, technology consultant and entrepreneur from the Bay area. He offers a blocklist for Facebook, via Github, and from the looks of it is an extremely thorough one indeed. Over 880 distinct domains, one heck of a long list.

dugan-github

It is constructed for use on Windows, Mac and Linux; all you need capability-wise is editing your computer’s host file. You can find this dream come true right here.

process-blocked-lsThank you sir.

Meanwhile, I’ve downloaded and edited this blocklist, removing the loopback IP addresses so it can be utilized easily within Little Snitch. You can find that here (plain text 26kb, which will not be updated past today). 100% credit goes to the previously mentioned originator, and copy/paste works like a charm.

MG signing off (safe from prying eyes)