Menu

Michael Gracie

A beginners’ guide to running blockchain full nodes using external storage devices

For those loath to run an established (read: huge) blockchain full node because they don’t want to reserve double digit percentages of their laptop’s soldered (additional read: not upgradeable) solid state storage for the cause, here are simple steps to do so by leveraging external drives. These instructions are for macOS, but Linux and Windows users should be able to derive their own setup from it.

First, let us note the location and size of comprehensive blockchain data stores as presented by both Bitcoin Core and Ethereum Wallet

Bitcoin Core default stores the Bitcoin blockchain under ~/Library/Bitcoin. There you will find several directories and files, including those for your wallet. That which we will want to deploy elsewhere is as follows:

1) ~/Library/Bitcoin/blocks at roughly 170 GB

2) ~/Library/Bitcoin/chainstate at roughly 3 GB

Ethereum is less unwieldly, but nonetheless a lot of data. It can be found under ~/Library/Ethereum; we’ll concentrate on the directories below:

1) ~/Library/Ethereum/geth/chaindata at roughly 80 GB

2) ~/Library/Ethereum/geth/lightchaindata at roughly 0.5 GB

Note that we are going to ignore other items under the main directories, including that representing keys i.e. wallets, wallet.dat in the case of Bitcoin and the /keystore for Ethereum. Why? While setups vary, my particular environment includes a laptop with FileVault2 activated, and a discrete backup drive encrypted AES-256 that receives a weekly clone of the former. Hence I always have two very secure copies of my keys and prefer to keep it that way.

Post-Identification

We are now going to move the big directories noted above to an external storage device. To keep everything organized, first create a “Bitcoin” and/or “Ethereum” directory on the device. Then copy the entire directories from above into those. Next, append “.old” to the original directory location names i.e the directories you copied from.

Now we will create symbolic links aka aliases to point the wallet applications to the new data store. Type the following into terminal …

For Bitcoin

ln -s /Volumes/XYZDRIVE/Bitcoin/blocks ~/Library/Application\ Support/Bitcoin/blocks

ln -s /Volumes/XYZDRIVE/Bitcoin/chainstate ~/Library/Application\ Support/Bitcoin/chainstate

For Ethereum

ln -s /Volumes/XYZDRIVE/Ethereum/chaindata \ ~/Library/Ethereum/geth/chaindata

ln -s /Volumes/XYZDRIVE/Ethereum/lightchaindata \ ~/Library/Ethereum/geth/lightchaindata

(where XYZDRIVE is your drive name)

Note that as the latter locations go for each blockchain, they are small in comparison to the others (not critical to move). Further, with the Ethereum light client data (still in beta), it may be advantageous to not move the lightchaindata directory, in case you are on the road and want to use your wallet in a jiffy.

Finale

Run Bitcoin Core and/or Ethereum Wallet. Assuming your previously stored blockchain(s) is not too far behind with syncing, it should start rolling right away. You can delete the “dot old” directories after you’ve confirmed all is right with the world.

MG signing off (to contribute to the stability of another network, without the storage headaches)

Grab and validate National Vulnerabilities Database updates

Here is a concoction to grab National Vulnerability Database feeds, specifically the Modified JSON and related metadata, then validate the reported sha256 hashes:

import urllib.request
import gzip
import hashlib

#json file
fileurl = 'https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.json.gz'
json_file = '/Users/laptopuser/Documents/Active/NVD/nvd-data/0326/nvdcve-1.0-modified.json.gz'
urllib.request.urlretrieve(fileurl, json_file)
json_file_open = gzip.open(json_file, 'rb')

#meta file
fileurl = 'https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.meta'
json_meta_file = '/Users/laptopuser/Documents/Active/NVD/nvd-data/0326/nvdcve-1.0-modified.meta'
urllib.request.urlretrieve(fileurl, json_meta_file)
json_meta_file_open = open(json_meta_file, 'r')

#get hash from meta file
for line in json_meta_file_open:
    li = line.split(':')
    if li[0] == 'sha256':
        ze_sha = li[1].strip('\n')
        print('Meta:', ze_sha)

#calc hash of file
sha256_hash = hashlib.sha256()
with json_file_open as f:
    for byte_block in iter(lambda: f.read(4096),b""):
        sha256_hash.update(byte_block)
    ze_hash = sha256_hash.hexdigest().upper()
    print('Calc:', ze_hash)
    
if ze_sha == ze_hash:
    print('MATCH')

json_file_open.close()
json_meta_file_open.close()

You will get output that looks something like this …

Meta: E3ECE7D603F091E68E60E68CD6E230A28BC9E23EFB7E9B8145E559D1910BE9A6
Calc: E3ECE7D603F091E68E60E68CD6E230A28BC9E23EFB7E9B8145E559D1910BE9A6
MATCH

No apologies for the basic code presentation, nor for using urllib.request.urlretrieve. Feel free to copy and paste into Jupyter notebook or PyCharm if syntax highlighting is desired; as the latter goes, I know that function is supposed to disappear but my application requires keeping a sizable rotation of NIST’s handiwork close by.

MG signing off (to grab and validate some more)

Betting the book will sell

… if author Annie Duke’s interview is any measure:

Made me go looking for Ms. Duke’s work. After checking Barnes & Noble stock, I was going for my keys when I realized I was actually preparing to get charged a roughly 50% premium for the trip. Too much vig for my taste.

MG signing off (having put my chips on the pass line instead)

Plugging mcrypt into PHP on macOS High Sierra 10.13: A Requiem

Since Apple’s desktop operating system was called OS X Leopard, I’ve been concocting little ditties on how to plug mcrypt functionality into PHP without complete recompilation and/or using a black box MAMP offering. The previous stop on that [eight year?] journey was with macOS Sierra. And it was/is the end of the line.

Queue list of excuses …

  • I don’t use PHPMyAdmin or Magento eCommerce (both major targets as PHP + mcrypt needs go)
  • I’ve been leaning on Linux VMs for relevant development work (trashing environments and starting over less often than ever)
  • I’ve taken a liking to Python, as it covers everything from number tumbling and data crunching to stupid little utilities for replacing website photos (dropping Anaconda in is a breeze)
  • And not really last but certainly not least, Apple’s latest release, High Sierra 10.13, came complete with a security-related flaw so ludicrous that I probably won’t upgrade until at least iteration two or three. It’s a trust thing.

For those who have already gained elevation in the States’ westernmost major range, all is not lost. If you run through the last set of instructions and replace any code that says “10.12” with “10.13” – along with identifying the proper XCode and PHP versions at the start – you should be ok.

Meanwhile, thanks for the good times.

MG signing off (because that’s all folks)

UPDATE 11/28/17: Like I said, lack of trust.

Three Storms

en·tro·py
/ˈentrəpē/
noun
1. a thermodynamic quantity representing the unavailability of a system’s thermal energy for conversion into mechanical work, often interpreted as the degree of disorder or randomness in the system.
2. lack of order or predictability; gradual decline into disorder.

Click the image for real time action (09/08/17 16:40 UTC reference)

MG signing off (not to be an amateur meteorologist, but simply ponder)

Right now in Houston

According to the Houston Food Bank

“Right now, one in five people in southeast Texas don’t have enough to eat.”

Probably more like one in three right now, so please click the logo below and give what you can. Link is via convio.net, and has been tested (i.e. I received a valid receipt directly from the HFB).

Medical supplies are also needed, and eBay/Paypal are matching donations up to a quarter million. You can contribute RIGHT HERE.

MG signing off (thinking Texans are pretty cool)

UPDATE: And here’s a Paypal link for Irma related relief -> https://www.paypal.com/us/fundraiser/107725536494624790

Eclipse Monitoring Station at DM79mq

Peak shadow, if not peak chatter …

MG signing off (to struggle with 5 WPM)

Let’s close this trip out on a good note

INTREPID PHOTOG (WITH THREE CAMERAS, FOUR RIGGED RODS, FIVE LENSES, AND A SIX PACK OF MODELO IN TOW): This looks like a good spot for closing out this trip. I need a really good shot, so don’t splash up the pool when you step in. You’ve been fishing that same fly all weekend … sure you don’t want to change it up? How ’bout a dropper?

YOURS TRULY (WITH ONE PARACHUTE ADAMS, TWO FEET OF 6X TIPPET, AND A THREE WEIGHT RADIAN IN HAND): I’m good.

INTREPID PHOTOG: If there is anything here, it’s gonna be sitting on the right edge. Deeper over there. Sun’s at your back, so watch your shadow. Be careful of that big log behind you. Wanna cast this rod?

YOURS TRULY: Got it. Nope.

Thirty seconds later …

INTREPID PHOTOG: Dude, where’s your fly?

YOURS TRULY: (Sigh)

Another minute goes by …

YOURS TRULY: Satisfied?

MG signing off (because it felt like work, but business was good)

Photo credit: James “You Really Need A Dropper” Snyder

Angler credit: Michael ” No I Don’t” Gracie

Programming the Yaesu VX-8DR on macOS, with the help of CHIRP and Valley Enterprises

In a word, easy.

The quad-band Yaesu VX-8DR is a nifty little radio, but in many respects it also suffers from feature overload. Hence menu pain. Even after several sessions of RTFM, one can still be lost. Programming via computer, please enter stage left on queue.

Unfortunately for Apple users, most pre-packaged programming utilities i.e. cable and software offerings cater to the Windows operating system and/or are designed for serial ports (think DE-9 connector). Those jokers with a spare MacBook Air lying around are almost SOL; they have to assemble their own tools instead.

With a little research it’s a moderately low effort endeavor. Even less if you just continue on here.

(more…)

Bring Blogger images into WordPress, the hard way

You migrate from Blogger to self-hosted WordPress. Your posts move over just fine, but for some reason (or another) your images forget their bus pass. Those pornographic stupid cat, hastily-prepared food, and trying-to-make-people-think-you-are-wealthy-instead-of-deep-in-debt vacation photos still show on the new site as they are properly referenced in the posts, but they actually remain on Google’s servers. You (or your client) don’t like that.

Meanwhile, the two plugins you found to solve this problem, Archive Remote Images and Cache Images, haven’t been updated in years. You take your chances anyway because you are lazy (if it is a personal site), or consistently over-promise and under-deliver (due to the impossibility of getting real work done at coffee shops). Either way, you must now hope you made a full site and database backup beforehand. If you did, you’re solution is now staring you in the face.

The script I concocted (shown after the jump) will get you a folder full of those images – with clean and pretty naming conventions – that you can upload to your wp-content directory, along with a SQL script to update links in your WordPress posts. Said programmatic wizardry dirty hack is written in Python – debugged using version 3.5.2 Anaconda custom (x86_64) on macOS 10.12.3 to be precise – and does rely on some SQL prep work. If you do not know Python, SQL and how to navigate directories while a terminal prompt blinks back, you have two choices: Google it (after determining what the definition of “it” is), or inquire about retaining me to do your work for you.

I’ll make the decision whether to continue easy too; if you cannot execute the following block of code sans assistance you are officially deemed “without paddle” …

SELECT * FROM `wp_posts` WHERE `post_content` LIKE "%blogspot%"
INTO OUTFILE '/home/dump/blogspotposts.csv'
FIELDS TERMINATED BY '|'

That look easy? Then proceed.

First, decide whether to run on your desktop (for future upload) or directly on server. Next, create a directory underneath where the script is located called /bspics. Lastly, make sure the directory the code is in is writable by all.

The code can be found here -> processblogspotimagelinks.py

Once you have changed the obvious stuff to suit your need, run it. Your /bspics directory will fill up with those images I promised – you can then place that entire directory underneath /wp-content – and you’ll also have a file called bsreplacescript.sql which you will run against your WordPress database to update image links in the associated posts.

Important [final] note: the coding was an iterative process, and some data analysis was done between steps in order to account for string possibilities encountered, generating clean file names, etc. It could be refactored, but wasn’t because 1) the end result works as intended and 2) removing those iterations would handicap attempts to modify it for a different data set.

MG signing off (to solve some not-so-commonplace problems)