Virtualization, Docker, OpenShift, and Poker

***This is part of an ongoing series I call "Mode 1 Storage Guy goes to a Mode 2 World."  I'm not an expert (yet), YMMV.***

Let's say you need to see how a poker website looks from CentOS, but you have a windows box.  What you do?  Probably install hyper-v (or vmplayer), download a .iso image of CentOS, and create a new vm out of that ISO.  Contained within that VM is every dll, every file, everything CentOS needs.

OK, let's say you invented a winning poker algorithm, and you want it to play 500 games of poker simultaneously.  You don't have enough hard drive space for 500 vm's - but you have enough for 500 containers.  So on that CentOS VM you install Docker and download 1 thin CentOS container image, and Docker knows what files on the CentOS VM each container needs to run.  Docker makes it super easy to download an image in docker: docker pull  is all.

Now let's say you're making tons of money and want to get 10,000 containers playing poker.  That many images won't fit in your computer - you need more computers.  So you buy a bunch of windows computers, and on each you install hyper-v, get a CentOS VM up, and install docker.  You have your 10,000 poker games running...but one computer dies, taking down the VM and 500 containers with it.  You lose all the chips you had in those 500 games.  What's more, people are starting to copy your algorithm, and you start losing!  You improve your algorithm, but how can you change the 9,500 containers in time?

So you cluster all your computers with hyper-v.  Good first step.  And then install OpenShift on all the CentOS VMs.  Now when a server dies, OpenShift is replicating each container to another VM and you don't lose the poker game.  What's more, every time you update the algorithm, you can use docker to create a new image and have OpenShift deploy it in place of the old algorithm container after each poker game.  

OpenShift, Trident, Docker, and SolidFire: Part 1

***This is part of an ongoing series I call "Mode 1 Storage Guy goes to a Mode 2 World."  I'm not an expert (yet), YMMV.***

We have a group of NetApp/SolidFire customers already live with OpenShift on SolidFire, which is very exciting but a bit scary too.  It's a bit scary because many of these clients went live without ever chatting with us!  This means they're running into issues like having to manually create hundreds of volumes, because they hadn't heard of our dynamic volume manager, Trident.

So we're partnering with RedHat to get a local OpenShift lab implementation tricked out with all the best SolidFire has to offer.  The goal is to get OpenShift running, then move on to containerized Elasticsearch and MongoDB and all sorts of other fun stuff.

Note: YOU DO NOT need NDVP in order to install/use Trident.  We do so here only for experience and demonstration purposes.

Here's the basic layout of the lab:  

1) 3 RHEL servers running as VMs in VMware (1 master, 2 other nodes)

2) NetApp Docker Volume Plugin installed

3) SolidFire for persistent storage (great API, all flash performance)

4) Trident for the automatic volume management

5) OpenShift Enterprise (instructions) as our container platform, installed on the RHEL servers.

6) OSE has several requirements, such as Docker as our container engine

The instructions for each of these are actually really good, so I'll just elaborate on a few things for this specific workflow.

• Start with the OSE requirements instructions.  You need to make sure you have the correct RHEL licensing to access the OSE repos or you'll hit a roadblock in a real hurry!
• Once you get to "Configuring Docker Storage" I recommend you detour over to the NDVP instructions, where you see "iSCSI RHEL/CentOS." 
• Complete those steps, then continue with the "Configuring Docker Storage" instructions and complete through the rest of the page.  I used option 1, presenting a LUN for docker to use as the storage pool.
• You can then install the NetApp Docker Volume Plugin (NDVP).  
• Note that this is where the storage expertise comes in.  You'll need to know the management and storage IP's for the SolidFire, you'll need to setup iSCSI on the RHEL servers, and you'll need to present targets from the SolidFire to your RHEL servers. 



Create an access group with all the IQNs.

• If you need help, I recommend this video.  You can find your iSCSI IQN with cat /etc/iscsi/initiatorname.iscsi and then create an access group on the solidfire for it:

• Don't forget to add the port (3260) to the iscsiadm discover command
• sudo iscsiadm -m discoverydb -t st -p 172.21.40.X:3260 --discover
• iscsiadm -m node -l to log into all available targets
• fdisk -l will show whether your mounts were successful and their device names
• Once your RHEL server has logged into each target, you're able to run netappdvp --config=/etc/netappdvp/solidfire-san.json &
• Don't forget the ampersand.  If you just run the command, it'll appear to hang but it's actually running.

NDVP Ready to Go!

And here we go, all three RHEL servers have the OSE prereqs and NDVP installed.  Here's our first NDVP-created volume!

Social Media Marketing

Here are a few things I've learned about social media marketing for a small business:

1) Facebook is the easiest and best platform by far.  However, you an expect your first couple posts to get great exposure among your followers, and then a big drop off.  It appears they try to incentivize you to pay for exposure by manipulating the algo - several others online have observed this as well.  The ability to target age, sex, and interests is very valuable.

2) Twitter has a ridiculously counter-intuitive UI.  It can easily take you 1-2 hours to get a single campaign up and running!  The best part of twitter ads is targeting people who follow a specific page: it's a ready-made demographic engine.

3) Linkedin is almost as complex as Twitter, but I think it's a hidden gem that is underutilized by pubs/restaurants.  Particularly the ability to target employees of specific companies is fantastic.  ROI is still to be determined...

4) Yelp is one of the most frustrating things in the world.  It's very expensive, low ROI, and very hard to get them to transfer your business to you.  It also has an algorithm that (for some mysterious reason) hides good recommendations but not bad ones, which can really damage your business.  My suspicion is they drive up bad reviews to incentivize you to pay for a premium membership, which allows you to pin a good review to the top.  I recommend this article for Yelp: http://marketingland.com/5-yelp-facts-business-owners-should-know-163054

5) Google My Business is just an awful UI.  It's almost as bad as Twitter.  But you absolutely have to focus here: google, and google maps, are the most important to your online presence.  I'm still experimenting with AdWords and site analytics to see what kind of results you can get.

6) TripAdvisor is pretty solid.  I haven't done any advertising here yet, but overall they've got the essentials. 

Trident for Kubernetes

Last week NetApp dropped a huge development in the emerging tech market.  It’s called Project Trident, and it makes storage easier for Kubernetes.  Backstory: Kubernetes (also called k8s) started at google, it’s software that manages containers.  Basically you take a bunch of linux servers with Docker installed and tie them together with k8s, and it manages which container should live where.  If a container dies, k8s replaces it with a new one, that kind of thing.  You can think of k8s as Vmware for containers, except free and open source.

Most clients aren’t using basic k8s, but rather enterprise versions of it like RedHat OpenShift or Apprenda for reasons like security, support, and version management.  Trident is compatible with any version of k8s, which means it solves a big problem for RedHat.

Trident is similar to our vCenter plugin only even smoother: it allows k8s to ask a storage array for an NFS share or iSCSI LUN instantly, plus all sorts of LUN management abilities.  Better yet it’s free, open source, and storage vendor agnostic.

You can find information about it here: https://github.com/NetApp/trident and  http://netapp.io/2016/12/23/introducing-trident-dynamic-persistent-volume-provisioner-kubernetes/

Pub Business: Part 1

My family and I recently took a massive risk - we purchased an Irish pub.  The pub is in downtown Grand Rapids, MI and had been struggling due to a lack of investment, management, and ownership neglect.  It was losing money and it wasn't hard to see why: service was awful, the building was dated, the beer was skunky, everything smelled.

Before

We ran the bar for 4 weeks after purchase to get a better understanding of the business, then shut it down for 6 weeks for renovations.  We re-opened 2 weeks ago and have endured the trial by fire, and I think we've come out the other side satisfactorily.  In this post I'd like to lay out our financial modeling, our business strategy, and the results so far.

Before

 I composed from the previous owner's tax returns an income statement. What it revealed is that draft costs and liquor costs are higher than industry standard, accounting for $7k in losses.  I frankly don't believe either of those numbers: the draft lines were 120ft long and poured pure foam, plus the staff (and their friends) were drinking for free.  I'd personally seen bartenders giving away booze!  I'm curious to anyone's advice on how (or why) the previous owner could have hid these costs.

The other thing this reveals is that food costs as a % of food sales are way too high: 43% in contrast with industry-standard 29%.  There's $35k disappearing there!

Before

Our business plan was pretty simple in concept:

1. $275k in loans and $225k in capital to purchase the bar and completely remodel it.
2. Tap talent for the NEW bar!
1. Design (logo, artwork, menu)
2. Construction (floors, ceiling, new bar, new cooler and draft lines, booths, etc)
3. Transform the staff and culture: many of the original team did not make the cut.  Anyone whose honesty we questioned or who did not have a customer-centric outlook quickly phased out of the bar, and the pub's great new look attracted better employees.  We took a zero tolerance policy to theft, sexual harassment, and set high standards for everyone.
4. Implement systems of quality: daily and weekly checklists to avoid crises and ensure tasks like ordering and cleaning were completed with accountability, plus replacing aging systems with modern ones with lower TCO.
5. Advertise: we are developing a strategy that combines media, community involvement, social media, and a ground game to fill our pub during non-peak hours.
6. Marketing: we tapped a mixologist and chef for parts of our menu.  Much of the whiskey, beer, and food selection was made by ownership.  And we made big pricing changes: I graphed the COGS and price of every item and created tiers of target margin, and we now only carry items that fit into those tiers.

Opening Day

Along the way, we've learned a few lessons.  One is that owning a pub is all-consuming: there are always a million things to be done, and you have to learn time management like none other.  Another is having to accept that in the bar business, you're simply going to have some unhappy customers, and you have to figure out what's in your power.  And last, uncertainty kills job creation.  We'd like to hire several more people, but it's so hard to predict demand and sales that we are just taking the work on ourselves.

After

I could cover a million large steps we took to drive sales, but let's leave that for Part 2.  So how about the results so far - are we going to be miserable failures?  For a baseline, I pulled daily sales data from the past two years (2014 sales were better than 2015 sales) and graphed it:

A couple of takeaways here: we are running at 247% of 2015 sales and 153% of 2014 sales.  We have not had a single day where we didn't exceed previous years!

We are wildly exceeding our expectations.  If this pace continues, we'll have plenty of money to hire a manager and let the pub stand largely on its own operationally.  That would give us the ability to focus on business development and efficiency.  I've created a calculator to model increased costs with increased sales, but there's a lot of uncertainty - will this trend continue?  We can't do 147% better than last year forever right?  How much of this is just a new-bar pop, vs how much can we create new business and beat the competition? 

After

Part 2 coming soon.  In the meanwhile, check out our website and facebook!  Or if you find yourself in west Michigan, stop in for a drink :-)

SolidFire vs EMC ScaleIO

Doing a bit of research and thought I'd write this down for posterity.  Disclaimer: I'm a SolidFire engineer.

SolidFire Pros:

1. All-Flash optimized
2. Global, inline dedupe and compression
3. Enterprise data services:
1. Snapshots, cloning, replication (sync, async, snapshot based)
2. Automation (openstack, api's, etc)
3. Vvols support and vmware integration
4. QOS
5. iSCSI or FCP

ScaleIO Pros:

1. All flash or hybrid
2. No dedupe or compression.  Never will have global dedupe.
3. Scales larger (1,000 nodes instead of 100 nodes)
4. Wider whitebox support
5. iSCSI only
6. Can live on top of a compute node, occupying free resources.

More conversation here: http://www.theregister.co.uk/2015/09/29/scaleio_solidfire_and_vsan/

http://virtualgeek.typepad.com/virtual_geek/2015/05/emc-day-3-scaleio-unleashed-for-the-world.html

SolidFire Architecture #1

 It's time I write a long-overdue overview of SolidFire: how it works, how it solves problems, and why service providers love it.  So here is Part 1!

First, SolidFire is not the solution to everything.  But it is the best in the world at what it does solve, which is why it won Gartner awards for the last two years.  Since this is an engineering blog, let's talk about how it works.

SolidFire hardware is regular servers with SSD's and no RAID, so you get commodity hardware prices and a truly software-defined architecture.  It protects data by writing it in two places using an algorithm we call Double Helix, and then earns space back with inline compression and global inline dedupe.  The global inline dedupe allows for much greater dedupe ratios than anything else on the market, because every block of data written is unique.  Other storage solutions have silos of dedupe, pools of blocks that are unique locally but duplicated many times throughout the environment.

The SolidFire robot

Today SF is iSCSI and FCP only.  When you create a LUN, SF chooses where in the cluster to place the data, removing the enormous complexity of we call the "placement question."  Let's spend some time on that: in most traditional storage environments, you have a couple of storage nodes that form capacity and performance silos.  When you scale out to 20 or 1000 nodes, your provisioning encounters a complex question: where do I place this data?  That spurs hours of performance and capacity analysis, trending and peaks vs average conversations.  On SF, the cluster does it for you.

It also solves the performance question that multi-tenancy brings by allowing you to provision performance.  Not just capacity, but performance!  SF does this by allowing you to set a minimum, maximum, and burst for each volume, guaranteeing a service level.

I've only scratched the surface on this one: we'll save the scale cluster model and more for the next blog post.