Bandcamp, Music and Android

Here's a quick walkthrough on how to download music from bandcamp.com into your android phone.  First, you'll need to download an unzip app - Easy Unrar is free. Find it in the app store (Google Play) and download it.

I'll throw red dots on these screenshots to make it easy for you to follow along.


















Next, get your download code and click on the link - for example, https://hopehymns.bandcamp.com/yum.  Input your code and click next.

Now click "here's how"

Click the second "Here's how", then select your file type (I used MP3 220, but I'm not a music expert so YMMV).   Then click download.

Great!  Your music is downloaded.  Now we need to unzip it.  Open Easy Unrar and click the "Download" folder.

If you've had your phone awhile, it might be tough to wade through all the downloads and find your album.  So click on the sort button on the upper right.

Choose sort by file size, large to small.

















Since your album is probably more than 100MB, it should be near the top.  Here you can see Hope Hymns, the album i want.  Check the box and click Extract.

Check this box and click extract as well.

 OK - good news and bad news.  Bad news is you're not done, good news is you're almost done.  Now we need Google Music to rescan and discover your unzipped album files.  Open up the Settings app and click on Applications.

 Now find and click Google Play Music

Almost done!  Click Storage


















!Important!  DO NOT click "Clear Data."  DO click "Clear Cache"

















Now reboot your phone, open up Google Play Music, and you're done!  Enjoy.

Pandas

This is just a running list of useful things about Pandas as I learn.

If you have data coming in from a .csv, use this: df=pd.read_csv('file.csv')

If your dataframe has strings and you want them to be numbers, use this: df.column = pd.to_numeric(df.column, errors='coerce')

If you have date/time in linux epoch format, you can convert using this: df['date'] = pd.to_datetime(df['date'],unit='s')

If you want to index on select columns: df.ix[:,:2]

df.describe() gives you min, max, avg, mean, percentiles, and std

df.groupby([column.other]).mean()

df[column.other == ]


Grab specific columns by name: df1 = df[['a','b']]

data.iloc[:, 0:2] # first two columns of data frame with all rows

this will square each cell, skipping the first column.  df.iloc[:,1:7]=df.iloc[:,1:7].apply(numpy.square, axis=0)

great link on managing jupyter: https://www.datacamp.com/community/tutorials/tutorial-jupyter-notebook

Great guide to pandas: http://www.lining0806.com:1234/pandas/Pandas%20DataFrame%20Notes.pdf

https://www.shanelynn.ie/select-pandas-dataframe-rows-and-columns-using-iloc-loc-and-ix/
https://www.youtube.com/watch?v=POe1cufDWFs

Some images you can ignore:

Rancher Setup Basics

A client asked recently how SolidFire can integrate with Rancher.  I had a few RHEL servers available, so I'm going to set up Rancher on RHEL.  Here are the first steps:

Install a supported version of Docker (align compatibility for Docker, K8s, and Rancher): curl https://releases.rancher.com/install-docker/1.12.sh | sh

sudo service docker start

sudo docker run -d --restart=unless-stopped -p 8080:8080 rancher/server

Alright, let's pause here.  What did we just do?  First, we installed Docker.  Docker is the software that enables you to easily download, create, run, and manage containers.  Next we made sure the docker service was running.  Last we downloaded a container that will run the Rancher software.  At this point, you should be able to reach rancher's gui at :8080.

So let's get K8s and Trident up!  First we need a place to deploy K8s.  Click Infrastructure | Hosts.

Click add host.

And then save

Then enter the IP address of the server that will function as a host for containers.  Follow the instructions to copy-paste the command into a console on your new host server.

Done!

https://docs.rancher.com/rancher/v1.5/en/installing-rancher/installing-server/#single-container

Why Rancher: http://rancher.com/beyond-kubernetes/ 

Understanding SolidFire Capacity

To calculate the effective capacity available, follow this formula:

• Error Threshold (#3) minus Used Capacity (#1) = Available physical space
• In the example below, 86.24TB - 48.41TB = 37.83TB
• Multiply Redupe Ratio * Compression Ratio, then divide by 2 (for Double Helix)
• In the example below, (1.81 * 1.79)/2 = 1.62:1
• Multiply the results of (a) and (b)
• 37.83TB * 1.62 = 61.29TB

*This calculation assumes the current rate of dedupe and compression will continue
*This calculates capacity until the Error Threshold is reached, not the Total Capacity.

From ActiveIQ, our cloud monitoring tool:

1. Used Capacity.  This is the capacity physically taken up on disk by data.  After dedupe, compression, and double helix occur, this number is the end result.
2. Warning Threshold.  This is an adjustable alert threshold to alert that you’re approaching the Error Threshold.
3. Error Threshold.  This is the point after which the system cannot rebuild the second copy of data after a node loss.  This is calculated by subtracting one node’s physical block capacity from the Total Capacity.
4. Total Capacity.  This is the raw physical space on disk.  In this example, 1.92TB * 9 SSD’s * 5 nodes = 86.42TB.

From the SolidFire GUI:

      5. Block Remaining.  This is calculated subtracting the Used Capacity (#1) from Total Capacity (#4).

      6. Block Capacity until Warning.  This is calculated subtracting the Used Capacity (#1) from Warning Threshold (#2).

***Bonus***

       7. In ActiveIQ, under Reporting | Cluster Efficiency, hover over the graph to view the dedupe and compression ratios.  In this lab system, 1.81 * 1.79 = 3.24:1

Setting up PowerShell for SolidFire

A simple Guide to setting up PowerShell for SolidFire.

1. Enable PowerShell
2. Download SolidFire PowerShell toolkit 
3. Unzip the toolkit
4. Navigate to PowerShell-master\PowerShell-master\Install\ and run SolidFire_PowerShell_1_3_1_4-install.msi 
5. Done!

https://www.youtube.com/watch?v=nvnpzP4-T5I

Trident in Action

Some screenshots of NetApp's dynamic storage provisioner for K8s!  In this case, we're using OpenShift on SolidFire.

OpenShift, Docker, and Elasticsearch

***This is part of an ongoing series I call "Mode 1 Storage Guy goes to a Mode 2 World."  I'm not an expert (yet), YMMV.***

There are already many good setup docs for Elasticsearch on OpenShift, like here and here.  So what I'm going to do is flush out the concepts so those instructions make more sense.

OpenShift

OpenShift has a concept of a project.  This is how they provide multitenancy.

Next is namespace, which is similar to a project. 

Next is the image.  An image is a pre-packed container probably with an application installed, like MongoDB or JBOSS.  You can create new images by installing an application into a container and saving that container.  The image concept is analogous to a VM Template: you keep it updated and deploy fresh containers from it.

A stream seems to be a set of evolving images: for example, when you download the latest CentOS, you're not asking for a specific version, just the latest version.  The stream is the set of successive images that you get the latest from.

Application.  This is the normal definition of an application, however in the context of containers, it's good to think about the application as separate from a container

The catalog is a view, a view of the images held in the registry.

The registry is merely what OSE calls the collection of images, held in a folder.

A Pod is the OSE equivalent of a container

A deployment is the mechanism that manages where, how many, and replication on pods for a given application.

Persistent Volume Claim (PVC): In OpenShift, this is a request for storage.  It can sit unfulfilled.

Persistent Volume (PV): When you create a PVC, and then match it to a real LUN/Export, you have a PV.

Don't panic if you see the word "cartridge."  That's just British for container.

Some notes:
openshift's gui is very foreign. Take some time getting used to it.
Unless you're an old linux admin, you're going to need to take the linux learning curve seriously. You'll really need to brush up on vi, ls, cat, curl, and wget.
The & symbol is your friend.  For any command that appears hung (but isn't hung, it's actually running) like "docker run ", throw an '&' at the end and I betcha it'll give you your prompt back. 
I had no luck connecting to the OpenShift GUI via chrome, but firefox worked fine (remember, https and 8443)
When deploying images from docker into OpenShift, don't underestimate the Pull Secrets.  OpenShift has to be allowed to pull the images from Docker.  This mechanism is intended to prevent unauthorized access of Docker images.

Declarative State Storage

***This is part of an ongoing series I call "Mode 1 Storage Guy goes to a Mode 2 World."  I'm not an expert (yet), YMMV.***

Anyone working on containers will have heard the term "declarative state" with regards to the number and type of containers.  Basically, it just means you have a platform that ensures you always have x number of containers.  In other words whenever a container dies/fails for any reason, the platform will recreate one in its place.  You're declaring how you want it to be from now on.

Say Kubernetes you tell it "I want 100 apache webserver containers" on a cluster of 10 hardware servers. It will spread 10 containers onto each server.  If you lose 2 servers, Kubernetes will re-create those 20 lost containers and spread them across the 8 remaining.*

This is very different from a Vmware-type mindset.  In Vmware you can say "deploy 100 vm's off this template," but Vmware doesn't watch the vm's, count them, and re-start them if they come down.  This is "imperative code" meaning "do what I tell you now," following a single order.

The existence of declarative state means storage has to change.  Previously, the storage layer just creates 10 LUNS, present them to the right servers, and you're done.  If the server dies, storage doesn't automatically present those LUNs to a new server, or delete them.  It sits, static, until someone comes and manually fixes it.

Luckily, in Kubernetes 1.5 there is a new concept of StatefulSets.  StatefulSets are all about enabling stability (i.e. persistence), and when combined with NetApp's powerful Trident connector it means you can create Declarative State Storage.  Which is exciting, because as far as I know I coined the phrase :-)

Declarative state storage means when a container is deleted, its corresponding volume is deleted.  When a container dies and is recreated, its corresponding volume is connected automatically.  When an application is scaled from 10 to 100 containers, the volumes are provisioned automatically.

More to come!

*In reality, Kubernetes creates a synchronous copy of each container.  So it doesn't actually re-create the container: it switches the secondary copy of the container to be primary (lowering the impact of interruption) and then re-creates a secondary copy elsewhere.  Cool stuff.

Virtualization, Docker, OpenShift, and Poker

***This is part of an ongoing series I call "Mode 1 Storage Guy goes to a Mode 2 World."  I'm not an expert (yet), YMMV.***

Let's say you need to see how a poker website looks from CentOS, but you have a windows box.  What you do?  Probably install hyper-v (or vmplayer), download a .iso image of CentOS, and create a new vm out of that ISO.  Contained within that VM is every dll, every file, everything CentOS needs.

OK, let's say you invented a winning poker algorithm, and you want it to play 500 games of poker simultaneously.  You don't have enough hard drive space for 500 vm's - but you have enough for 500 containers.  So on that CentOS VM you install Docker and download 1 thin CentOS container image, and Docker knows what files on the CentOS VM each container needs to run.  Docker makes it super easy to download an image in docker: docker pull  is all.

Now let's say you're making tons of money and want to get 10,000 containers playing poker.  That many images won't fit in your computer - you need more computers.  So you buy a bunch of windows computers, and on each you install hyper-v, get a CentOS VM up, and install docker.  You have your 10,000 poker games running...but one computer dies, taking down the VM and 500 containers with it.  You lose all the chips you had in those 500 games.  What's more, people are starting to copy your algorithm, and you start losing!  You improve your algorithm, but how can you change the 9,500 containers in time?

So you cluster all your computers with hyper-v.  Good first step.  And then install OpenShift on all the CentOS VMs.  Now when a server dies, OpenShift is replicating each container to another VM and you don't lose the poker game.  What's more, every time you update the algorithm, you can use docker to create a new image and have OpenShift deploy it in place of the old algorithm container after each poker game.  

OpenShift, Trident, Docker, and SolidFire: Part 1

***This is part of an ongoing series I call "Mode 1 Storage Guy goes to a Mode 2 World."  I'm not an expert (yet), YMMV.***

We have a group of NetApp/SolidFire customers already live with OpenShift on SolidFire, which is very exciting but a bit scary too.  It's a bit scary because many of these clients went live without ever chatting with us!  This means they're running into issues like having to manually create hundreds of volumes, because they hadn't heard of our dynamic volume manager, Trident.

So we're partnering with RedHat to get a local OpenShift lab implementation tricked out with all the best SolidFire has to offer.  The goal is to get OpenShift running, then move on to containerized Elasticsearch and MongoDB and all sorts of other fun stuff.

Note: YOU DO NOT need NDVP in order to install/use Trident.  We do so here only for experience and demonstration purposes.

Here's the basic layout of the lab:  

1) 3 RHEL servers running as VMs in VMware (1 master, 2 other nodes)

2) NetApp Docker Volume Plugin installed

3) SolidFire for persistent storage (great API, all flash performance)

4) Trident for the automatic volume management

5) OpenShift Enterprise (instructions) as our container platform, installed on the RHEL servers.

6) OSE has several requirements, such as Docker as our container engine

The instructions for each of these are actually really good, so I'll just elaborate on a few things for this specific workflow.

• Start with the OSE requirements instructions.  You need to make sure you have the correct RHEL licensing to access the OSE repos or you'll hit a roadblock in a real hurry!
• Once you get to "Configuring Docker Storage" I recommend you detour over to the NDVP instructions, where you see "iSCSI RHEL/CentOS." 
• Complete those steps, then continue with the "Configuring Docker Storage" instructions and complete through the rest of the page.  I used option 1, presenting a LUN for docker to use as the storage pool.
• You can then install the NetApp Docker Volume Plugin (NDVP).  
• Note that this is where the storage expertise comes in.  You'll need to know the management and storage IP's for the SolidFire, you'll need to setup iSCSI on the RHEL servers, and you'll need to present targets from the SolidFire to your RHEL servers. 



Create an access group with all the IQNs.

• If you need help, I recommend this video.  You can find your iSCSI IQN with cat /etc/iscsi/initiatorname.iscsi and then create an access group on the solidfire for it:

• Don't forget to add the port (3260) to the iscsiadm discover command
• sudo iscsiadm -m discoverydb -t st -p 172.21.40.X:3260 --discover
• iscsiadm -m node -l to log into all available targets
• fdisk -l will show whether your mounts were successful and their device names
• Once your RHEL server has logged into each target, you're able to run netappdvp --config=/etc/netappdvp/solidfire-san.json &
• Don't forget the ampersand.  If you just run the command, it'll appear to hang but it's actually running.

NDVP Ready to Go!

And here we go, all three RHEL servers have the OSE prereqs and NDVP installed.  Here's our first NDVP-created volume!

Social Media Marketing

Here are a few things I've learned about social media marketing for a small business:

1) Facebook is the easiest and best platform by far.  However, you an expect your first couple posts to get great exposure among your followers, and then a big drop off.  It appears they try to incentivize you to pay for exposure by manipulating the algo - several others online have observed this as well.  The ability to target age, sex, and interests is very valuable.

2) Twitter has a ridiculously counter-intuitive UI.  It can easily take you 1-2 hours to get a single campaign up and running!  The best part of twitter ads is targeting people who follow a specific page: it's a ready-made demographic engine.

3) Linkedin is almost as complex as Twitter, but I think it's a hidden gem that is underutilized by pubs/restaurants.  Particularly the ability to target employees of specific companies is fantastic.  ROI is still to be determined...

4) Yelp is one of the most frustrating things in the world.  It's very expensive, low ROI, and very hard to get them to transfer your business to you.  It also has an algorithm that (for some mysterious reason) hides good recommendations but not bad ones, which can really damage your business.  My suspicion is they drive up bad reviews to incentivize you to pay for a premium membership, which allows you to pin a good review to the top.  I recommend this article for Yelp: http://marketingland.com/5-yelp-facts-business-owners-should-know-163054

5) Google My Business is just an awful UI.  It's almost as bad as Twitter.  But you absolutely have to focus here: google, and google maps, are the most important to your online presence.  I'm still experimenting with AdWords and site analytics to see what kind of results you can get.

6) TripAdvisor is pretty solid.  I haven't done any advertising here yet, but overall they've got the essentials. 

Trident for Kubernetes

Last week NetApp dropped a huge development in the emerging tech market.  It’s called Project Trident, and it makes storage easier for Kubernetes.  Backstory: Kubernetes (also called k8s) started at google, it’s software that manages containers.  Basically you take a bunch of linux servers with Docker installed and tie them together with k8s, and it manages which container should live where.  If a container dies, k8s replaces it with a new one, that kind of thing.  You can think of k8s as Vmware for containers, except free and open source.

Most clients aren’t using basic k8s, but rather enterprise versions of it like RedHat OpenShift or Apprenda for reasons like security, support, and version management.  Trident is compatible with any version of k8s, which means it solves a big problem for RedHat.

Trident is similar to our vCenter plugin only even smoother: it allows k8s to ask a storage array for an NFS share or iSCSI LUN instantly, plus all sorts of LUN management abilities.  Better yet it’s free, open source, and storage vendor agnostic.

You can find information about it here: https://github.com/NetApp/trident and  http://netapp.io/2016/12/23/introducing-trident-dynamic-persistent-volume-provisioner-kubernetes/