Thursday, June 16, 2016

SolidFire Architecture #1

 It's time I write a long-overdue overview of SolidFire: how it works, how it solves problems, and why service providers love it.  So here is Part 1!

First, SolidFire is not the solution to everything.  But it is the best in the world at what it does solve, which is why it won Gartner awards for the last two years.  Since this is an engineering blog, let's talk about how it works.

SolidFire hardware is regular servers with SSD's and no RAID, so you get commodity hardware prices and a truly software-defined architecture.  It protects data by writing it in two places using an algorithm we call Double Helix, and then earns space back with inline compression and global inline dedupe.  The global inline dedupe allows for much greater dedupe ratios than anything else on the market, because every block of data written is unique.  Other storage solutions have silos of dedupe, pools of blocks that are unique locally but duplicated many times throughout the environment.
The SolidFire robot

Today SF is iSCSI and FCP only.  When you create a LUN, SF chooses where in the cluster to place the data, removing the enormous complexity of we call the "placement question."  Let's spend some time on that: in most traditional storage environments, you have a couple of storage nodes that form capacity and performance silos.  When you scale out to 20 or 1000 nodes, your provisioning encounters a complex question: where do I place this data?  That spurs hours of performance and capacity analysis, trending and peaks vs average conversations.  On SF, the cluster does it for you.

It also solves the performance question that multi-tenancy brings by allowing you to provision performance.  Not just capacity, but performance!  SF does this by allowing you to set a minimum, maximum, and burst for each volume, guaranteeing a service level.



I've only scratched the surface on this one: we'll save the scale cluster model and more for the next blog post.

Sunday, May 29, 2016

BlackPhone 2 Review

I've been excited about the Blackphone 2 for quite some time and finally switched over to it.  Partially because I want to make it harder for criminals, companies, and the government to intrude on my privacy, partially because I want to encourage the tech industry to implement smart security measures.  Why in the world does pandora demand access to my iPhone's calendar?  This has gotten out of hand.

Unfortunately, the Blackphone 2 is not ready for prime time.  Here's a short list of why.
  • Any attempt to update the OS is met with "Download failed" with no debug data for why it failed.
  • It's freaking huge.  One handed, fully 50% of the screen is out of your thumb's reach.  And the weight of it is killing my wrist.  And in a world where I need map on my phone while driving or coffee in my hand while texting, a phone this big is basically worthless. 
  • The touch screen is really inaccurate.  I'll tap the same icon 5-10 times before it's registered and acted upon.  Sometimes I'll tap an button in an app and SilentOS will pass the tap back to the home screen, opening a completely application
  • Google maps keeps failing to find my location, even though I've given it permission. 
  • Even though I blocked Facebook from accessing my location, it is able to access my location via Google Play Service.  So what's the point, Blackphone?  It's not granular permission if you're giving them a giant back door.
  • The volume/power combo for taking a screenshot is very hit or miss.
  • No built-in visual voicemail.  Gotta download your carrier's app.
  • Notifications take a TON of configuring.  Let me put it this way: new applications should not be able to display full notifications on a locked screen.  For example, Whatsapp or Facebook Messenger shouldn't default to publishing the entire text.  A private-by-design phone should at least only show the name of the contact, but preferably only show that a message exists.
  • The phone's usability is 5 years behind an iPhone.  Some examples:
    • My mom texted me a picture and I wanted to send it to my sister.  There's literally no way to copy/save a picture in BlackPhone's text app. 
    • Highlighting text is a real, real pain.  But copying it is impossible, because the copy button is at the very top the screen, 4 inches away from your thumb.
    • The triangle-circle-square buttons at the bottom of the screen are endless torture for me.  Apps like outlook have a set of buttons at the bottom of the screen and it's completely counter-intuitive to have two layers of buttons.  I accidentally reply to emails several times a day. 
    • To open the camera from a lock screen on an iPhone, you swipe upwards.  To do the same on the Blackphone you have to unlock the phone.  This is a big deal because I frequently want to record events transpiring around me, and I might miss it while unlocking the phone.
  • It takes forever to reboot.
  • It's noisy.  Why did I just get a notification that I took a screenshot?  I know I took a screenshot.  Why did the keyboard icon just appear in the upper right?  I know I'm using the keyboard. 
Some good things:
  • The display is great
  • Tethering works great
  • The wifi by GPS feature works great
  • I love the granular control "security center" where it works.  
  • Apps seem to be compatible and work just fine usually.
Google play services (as mentioned before) kills the whole point of the phone by handing control over: https://charlescarrollsociety.com/2016/02/03/silentcircle-blackphone2-is-not-secure-patriot-privacy-privacyproject-review-part-3-of-3/

I'm going to be asking Silent Circle if there's a new, smaller version coming out in the next couple months.  If not, I'll probably give this to some kid to use as a tablet.

Friday, May 6, 2016

Content Delivery Networks

I've been doing a bit of research into both Data Science and the hyperscalers and a few things have struck me.  One is that Google Fiber is incredibly strategic for Google Platform.

The ability to ensure QOS and cache by controlling the last mile of delivery is a huge advantage.  Everything AWS does is dependent on the telcos, because all of their wonderful technology is designed to deliver something (data, a website, a database) from AWS to somewhere, or from somewhere to AWS.  And that something almost always goes through Verizon, ATT, Comcast, Time Warner, etc.

So one of the most innovative, client focused, fast-paced tech giants is completely dependent on the most change resistant, oligopoly, entrenched companies in the US.  ATT and Verizon do some good business in the private-cloud enterprise space, but I've seen up close and personal that their company culture is bureaucratic as it comes.  And of course, there is cost involved: as all of us commercial users of ISPs know, the oligopoly always gets its money.

That's why Google Fiber is so forward thinking.  They're going to leapfrog AWS on this one, and if Google succeeds in laying a network quickly, in 15-20 years it could give Google Platform a killer competitive edge.  

Wednesday, March 16, 2016

SolidFire and ONTAP

I had a reseller ask last week "Now that NetApp bought SolidFire, are they going to kill all-flash FAS?"  My answer: not on your life. 

NetApp has sold tens of thousands of all-flash FAS (AFF) systems, which run CDOT, our flagship operating system.  It's a great product that enormous enterprises (and governments) are spending a billion dollars a year on: there's no way we'd back down from continuing to invest in R&D there.  

Besides that, SolidFire has a completely different architecture than AFF.  One way to understand it is that AFF's architecture starts with smaller building blocks.  Here's what I mean:

  1. AFF dedupes each volume individually: SolidFire dedupes the entire cluster.
  2. AFF protects each disk using RAID: SolidFire protects each node using two copies of everything.
  3. AFF puts QOS on each volume: SolidFire shows you whether your QOS promises exceed the cluster's ability.
  4. AFF deals with node failure by having a redundant partner take over: SolidFire deals with node failure by having ALL the other nodes pick up the slack.




These are different architectures, which solve different needs.   I thought this was a great overview of SolidFire as well:
http://www.virtualtothecore.com/en/solidfire-a-quality-storage/

Tuesday, February 23, 2016

Performance Archive Part II

More on performance archiving: it collects data at the highest granularity possible for each statistic, often 1 second.  The time range of each dataset is up to 6 hours long, and each node preserves its own set of this performance data. 

There is nothing like this in 7mode (rolling perf data collection) on-box.  You’d have to run perfstats. 

NetApp has tools to analyze the data, but they are not customer facing.    You provide the case number in the command line and it auto-uploads.


You can specify the exact time window you’re looking for:
cs001::> system node autosupport invoke-performance-archive -node cs001-pn01 -start-date "02/22/2016 8:00:00" -end-date "02/20/2016 14:00:00"


There is minimal impact on the netapp system of running this – it already runs in the background and keeps up to 28 days of data.

Friday, February 19, 2016

SolidFire

SolidFire is a "make private cloud easy" solution primarily designed for service providers.  It's a "born in OpenStack" all-flash whitebox solution that aims to be stupid-easy to deploy and manage.

The goal for SolidFire is not to be the fastest, the most resilient, or the most features.  It aims to answer one question, best in class: "How do I easily deploy Storage as a Service?"  You can see this in their design choices:
  1. Because this is a product service providers sell, they're flash only, have required QOS policies, and skip all the management tools, leaving that to OpenStack.
  2. Because they use two copies of everything instead of RAID, they achieve node level resiliency and skip expensive hardware and software, using inline dedupe/compression to recover the space delta.  This also spreads performance requirements across the entire cluster.
  3. Because they expect you'll be deploying a single configuration thousands of times, they support only 1 protocol and have very limited configuration options.
  4. Because this is for a cloud, not a single-purpose, the cluster (up to 100 nodes) auto-grows when you add a new node and recovers quickly when you lose one.
A few technical details:
  • Platform today is Dell servers.  Now that Dell owns EMC, it'll probably convert to Cisco.
    • 10 drives per node
    • SF2405: 5-10TB and 50k IOPS
    • SF4805: 10-20TB and 50k IOPS
    • SF9605: 20-40TB and 50k IOPS
    • SF9010:  20-40TB and 75k IOPS
  • Features:
    • Inline dedupe and compression
    • For QOS you can set min, max, and burst limits.
    • Mix any node platform
    • You can hot remove nodes
    • iSCSI, FCP (with a gateway device)
    • native snapshot capability and can backup to any Amazon Web Services S3 or OpenStack SWIFT-compatible API. 
  • Under the hood:
    • Nodes are connected via 10GbE over your shared network.  Not a private intracluster network.
    • “All connections for a particular LUN presented to storage go back to the primary node for that LUN. IE: multipath doesn't help you weather a failover. They're dependent on long iSCSI timeouts to give them time to fail a node and redirect traffic.”
  • Performance and QOS: http://www.solidfire.com/resources/provision-control-and-change-storage-performance-on-the-fly
  • Node Loss Demo: http://www.solidfire.com/resources/demonstration-of-solidfires-automated-self-healing-ha
SolidFire wins Gold in the Storage magazine/SearchStorage.com 2015 products of the Year Storage Systems: All-Flash Systems category.  http://searchsolidstatestorage.techtarget.com/feature/SolidFire-SF9605

CDOT Tip: Performance Troubleshooting

NetApp is engineering simpler, more elegant tools for our clients to manage their large technology deployments.  We’ve found recently that clients are relatively unaware of one tool that can make your life much easier when you are investigating a performance issue: Performance Archives.

Performance Archives has existed in ONTAP for a long time, but beginning with 8.3 (released November 2014), the payload was updated to specifically enable diagnostic use cases, trending use cases, etc.  In 8.3, there is a command “system node autosupport invoke-performance-archive” which allows customers (HTTPS enabled) to ship back up to 6 hours worth of data at a time, collected a much higher resolution than off-box PerfStat ever did (per-second for many counters) AND allows you to “go back in time” up to 28 days, depending on customer configuration. 

The tool we recommended pre-8.3, PerfStat, will continue to function through 8.4 per current plan but we recognize it is post-failure collection mechanism, which is not ideal.  In other words, after you run into a performance issue, you have to set up PerfStat to run and wait for that issue to recur.  Performance Archives gives you the ability to instantly look back several hours and catch what the problem in the act.

We’re also making big improvements to Performance AutoSupports:  We are focused on efficiently streaming real-time performance information to our cloud and making it easier to access this content within the client-facing NetApp ASUP infrastructure. This will allow for our customers and NetApp support engineers to do trending, analytics, diagnostics and more.


And of course OnCommand Performance Manager remains your go-to tool for retaining, graphing, and machine learning analysis for your entire NetApp footprint.

Read more here: https://kb.netapp.com/support/login/p_login.jsp?page=content&id=3014366&locale=en_US