Transcription (thanks Margaret) from the February 7, 2008 Data Storage/Management meeting. The recording was pretty rough, so Margaret had to do a lot of interpreting/guessing. Hopefully there is still good, useful, information here. Not really expecting that anyone will directly read much of this, but it's hear for searchability.
--
Data Storage Management Meeting/Discussion
February 7, 2008
Speaker: Ken Celecia
First, I’d like to tell you a little bit about Fairway Consulting Group, who we are, what we do, and then I’ll dive right into the infrastructure that we’ve constructed. I also have some hand-outs and they are available electronically if I can get your cards, I’ll send them to you via email. Let me briefly touch on who we are and what we do, we’ve been knocking around with Tony and the Church IT Roundtable group for about a year now, be a year in April. Fairway Consulting Group is an IT Consultancy that specializes in the virtualization space. We are a 7-year old company with headquarters in Fort Lauderdale, Florida and the sales office here in Atlanta. I’ve been with the company for about one year now. I did not come from the IT space, I came from another area, the fellow that hired me, we worked together some years ago. So technically, I’ll do the best I can, but if I need help, we’ll dial up my colleague and we’ll get technical, so we’ll be able to do the sales side, I do the sales side, but when we get to the technical piece, if we get a little bit too far out on the ice, we’ll give Jeff a call, he is ready to accept any questions we might have.
Primarily we are VMware and a company called Data Core which does storage virtualization, some people are familiar with Data Core, some people aren’t. Data Core does essentially the same as VMware does, they virtualize your storage much like VMware virtualizes your computer environment. If you have any particular questions around Data Core, we can get to that.
Last year, we constructed, or built-out, and upgraded our infrastructure that we have. Essentially, what we have is about a half-million dollar infrastructure that is primarily made up of Sunfire, X4100, M2 2.8 gig servers and we have about 40 terabytes of free space primarily on Promise Technology’s hard-drive arrays. It runs our production, it runs hosting for other companies and essentially what we have and what we are offering within this particular infrastructure are two different products. One is a hosted virtual services infrastructure whereby you all can run virtual machines out of our location, or a disaster recovery business continuity [Time Stamp00:03:30] platform that really was the genesis of Tony’s and our discussion and the discussion with the Church IT Roundtable about a year ago. The actually facility is a host.net facility. We have two diverse and redundant 20 amp circuits into our infrastructure, we have two diverse, and when I say diverse for the electrical piece, it’s off two different grids, two diverse and redundant 4 meg to 20 meg IP circuits as well, we’ve got 12,000 gallon diesel back-up along with that and we actually have some of the power companies down in Florida and some of the fuel companies are actually within the host.net facility, so you know they are not going to go down either.
The infrastructure is on the second story of the building, it is a category 5 rated structure, triple-DES authentication to get in, badge, there’s a guard and biometric scan. Our infrastructure also has a 50-ton cooling system that is available to it, smoke and water detection, FM 200 fire suppression, so the host.net people are very proud of the infrastructure that they’ve build there, and host.net also has different locations around the country as well, so as we grow, our particular infrastructure will also maybe think about building out different where they have their space.
So let’s talk briefly about the disaster recovery business continuity platform, and I’m not going to bore you with a 20 minute discussion because that’s just not what I do. Essentially, we will come into anyone’s infrastructure that is interested in protecting their servers or protecting their data and take virtualized images of your business critical servers. We virtualize them. We’ll also do a data extraction of your business critical data, we’ll upload that to our facility down in the host.net facility. Sometimes we can do it over the wire, sometimes it requires onsite presence to do that. Either way we can get it done. Then after it is uploaded to our facility, [Time Stamp00:06:43] we do a full sync of your data, which basically we receive the block level changes that have occurred in transit, then you pass the block level changes on a continual basis to us to your data structure and in the event that something happens in your infrastructure, you give us a call and declare an emergency, and at that point, you repoint your DNS and we wake up your servers and your protected space on our infrastructure and depending up on how many servers you have protected with us, we can get you back up and running in our facility within minutes to hours, it depends on how big the environment is.
So rather than building out your own DR business continuity infrastructure wherever you want to do that, we have the infrastructure, we maintain it, we maintain the servers and the storage, we make sure all the latest revisions, etc is up and running and up to date, and in the event that something bad happens, you can work directly out of our location.
But it is not just for disaster recovery business continuity, we have the ability to move these workloads back and forth. We are talking with a company in Long Island NY who is going to be moving their offices to a data center in South Carolina, and what we’ve proposed to them is to go ahead and perform the virtualization piece for them, then we can help them as they physically move their infrastructure, we can virtually move the infrastructure down to the location in our hosted facility, because they are just like everybody else, these are business critical servers, they can’t go down it’s gotta be 24/7, etc. We can provided them a way to keep these particular servers up and running while they are putting stuff in boxes and moving it physically to the South Carolina location, then once they’ve got everything set up in the South Carolina location and powered up, we can basically move that workload back to their location if that’s what they’d [Time Stamp00:09:31] like to do and they are up and running as if nothing happened.
It basically turns your moving day from Friday at 5:30 to Sunday at 4:30, to whenever you want to do that, and people will be able to connect to their servers and they can do whatever business critical stuff they need to do just to get through their regular working day. If you are in the path of a storm, it is the exact same thing. We can actually move those workloads once we have you virtualized and in our space, and you can secure your particular site, the servers and the infrastructure runs on our infrastructure that you have secured with us, and once the storm passes and everything gets cleaned back up, we can just move the workload back.
It’s pretty advanced, revolutionary, the people we talk to think it can be of benefit to them.
It’s about everything the Church IT Roundtable was looking for way back when, and I think you guys kinda pushed us into thinking out of the box in constructing this kind of infrastructure, so we’ve got it constructed, it’s ready to go, if it needs to be expanded, we expand it, we’ll buy another server, we’ll buy more disks, we’ll buy another Promise array, it’s all right there and we maintain it and work on it on a 24/7 basis. We’ll even give you a portal where you can look into your particular infrastructure, if you need to be satisfied that there is a heartbeat on the other side, we can do that. We also have extra licensing and extra software where if you guys wanted to test it out before you jump in the pool, we’ll be happy to set something up. So, that’s the brief outline of what the infrastructure is, how it works from a sales perspective. I’ll do my best answer the technical questions. I can answer the first question, “How much bandwidth?” Well it really depends on what you’re doing on your end, if it’s a SQL server that you’re really crunching a lot of numbers on, etc, you probably need a larger pipe to get to the Internet, but if it’s email or something of that nature, we might be able do to it given your present circuit. Every night, we go into all of our hosted and our disaster recovery business continuity customers and get a complete scoop-up of what the environment is and take an image of it that is transferred to tape and then moved to an offsite location, so there’s back-up to the back-up. The infrastructure is in a full HA mode on both the VM ware side and the Data Core side which is your compute and your storage environment. I had pictures, I’m told there are pictures of it but it’s not really a lot to look because it is fully virtualized, it’s in a couple of racks. The beauty of VM ware and Data Core as you know is you can squish everything down to a very small footprint and that’s it. People can’t believe that we can run so many virtual machines and so much virtual storage in such a small footprint but that’s the way of the world nowadays. That’s my talk. I have a very unsophisticated PowerPoint presentation, I’m tired of the flashy slides, I think I stand out more with my plain and simple slides.
What questions can I answer? Do I need to get Jeff on the line?
Question
Where is the data center located? Is that in Florida?
Ken
Boca Raton, Florida. It’s Host.net is the name of the outfit.
Tony Dye
Very early in the talk you said you had 50 terabytes of free space, do you mean “free” or “available”?
Ken
Available! And a little bit more than 100 virtual servers available, and that’s in my unsophisticated PowerPoint.
Question
How [Time Stamp00:15:16] long do you keep the back-ups? And if we have a failure on our side, can you restore from one of your back-ups something we lost on one of our local servers?
Ken
Yes, we can. That’s a Jeff question, we use Microsoft Data Protection Manager as well, and we can talk about backing up the back-ups and how those things are treated, I’ll get that.
Tony
On that same subject, I’m interested in your security model for the things you take offsite. How are they secured going offsite, how are they secured at the secondary, and where is the secondary site?
Ken
Ok, we’ll ask Jeff that one as well.
Tony
Someone asked the question – since Boca is a hurricane area, what’s your uptime, resolution, back-up, all those things?
Ken
Category 5 rated building, which that doesn’t mean diddly to you because you sit in Atlanta and don’t get hurricanes here. On the second floor, from the water perspective, we have not constructed anything up here where we can just basically scoop up the entire [?] and move it up to here but we plan on doing that. We can ask Jeff about procedures, what-ifs, in the path of a hurricane. I believe this is a newly-constructed building built post-the most-recent hurricane, not Katrina, but the one before that.
We rarely go into the facility ourselves, most everything we do is done remotely as well.
Tony
A 5 building and having the power companies there, those people don’t evacuate, those are the people who stay behind and keep stuff going.
Ken
Host.net if I’m not mistaken, has other hosting facilities and what I would like to do is to build another one further inland, an infrastructure where I can just move the entire thing up here or to Oklahoma, less tornado country, but somewhere else that is safer that is not in the direct path of a storm. Because [Time Stamp00:19:04] you here Category 5, you hear all of this and you say, “Well what if?” Nobody could predict Katrina either, so absolutely valid question, and Jeff has actually been to the facility, I’ll be going within the next couple of weeks to actually see, so I can’t speak to what it physically looks like but I know a little bit about how the constructed it and why they constructed it, but Jeff can fill that out for us.
Tony
The obviously related question is the building stays up, you got power in the building, but where are all the fibers that run to it, how do you keep them going? And obviously New Orleans, they had infrastructure of some sort the day after or the day of, something was there, so there is some fiber that stays hot somehow.
Tony
Something was there so there is some fiber that stays hot. If you’re looking at this from a different perspective, if you’re running out of there data storage, it’s a whole different thing then if you their data center is your hot back-up center, cause what does it take to take Boca out and Atlanta out at the same time?
Ken
Big! Then do we care anyway?
Tony
Now the hurricane that takes Boca out and spawns tornadoes up here, that’s where you get into, maybe there is something to it.
Question
How do you go about testing the infrastructure? Every six months?
Ken
We do have a testing procedure, but we’ll have to defer that to Jeff, but I know that we test.
Speaker
It’s just like back-ups, you want to make sure to test your back-ups so that when everything dies,
Ken
Absolutely.
Question
What kind of infrastructure is required on our side? Do we have to have VM ware, do we have to have Data Core? Ya know, what pieces do we need on our end?
Ken
The interesting thing about the requirements on your side is if it can be virtualized, we can take it and put it in our infrastructure. There’s no driver requirement or no type server requirement that I’m aware of. I’ll have Jeff confirm that, but the way we did it was “from us to any and from any back to us.”
Speaker
So if we had 8 or 10 servers sitting in our rack, wherever our company was, you could take those and make them into hot stand-bys down in Florida, which would be virtual even though ours are physical.
Ken
That’s right.
Tony
Similar question, if I’m using you as my offsite back-up, and forget me having a major failure, let me have a little failure, like somebody deleted a file or somebody blew up their Exchange mailbox, how well can I recover just that piece across the wire? I just want to retrieve some chunk of data, and granularity of Exchange and SQL and Sharepoint and all that kind of stuff would be a huge plus if you’ve got a good tool.
Ken
Yes, I think that goes back into Data Protection Manager, what we can do with that.
Tony
So you’ve got local protection and [?] protection probably.
Ken
Right. So what if it is a minor emergency?
Tony
Yeah, a stupid thing, as opposed to a ‘something blew up’ thing. I’m sure nobody else ever has those stupid things happen.
(Long pause)
Ken
Is it plausible, is it believable, does what we’re saying make sense? Does it pass the smell test with you guys?
(conversation going on in the background, I can’t hear it)
Tony
So all that to say, the tool that’s used for back-up, that granularity, I almost feel like is a foundation to where we go with this. For me, just back-up and offsite back-up managed would make this a win. Everything is a gain on top of that. Disaster recovery and business continuity, I think they are two different things, but you don’t have business continuity without disaster recovery.
Ken
It’s almost like disaster recovery being something really bad, business continuity would be like the stupid erasing of a file.
Tony
Disaster recovery is the building blew up, business continuity is they cut all the cables. Can you keep me going in both cases?
Ken
Well, to keep the conversation (lost sound)
Tony
But can I go home and get to it? Loss of connectivity, can I say, “We’re gone, switch to the data site”? Whatever is there is better than whatever is not available here.
Especially if you are an e-commerce, web-hosting type thing. Need to very dynamically switch back and forth or even have hot redundancy.
Speaker
Talking about business continuity, what about your phone service, because your phone is even more important than your computers.
Ken
We have knocked around the idea of hosted voice-over IP, we’ve identified a gentleman who is familiar with the Cisco brand products, we’ve not moved on that because most folks are data and server concerned and if your phone gets knocked out, you pick this up (must be holding a cell?). But if you are an e-commerce site with phone calls coming in with customers that have to be answered, we’ve thought about the voice over IP as well, as part of the infrastructure and from our view, it can pretty easily be installed and enabled.
Let me see if I can get Jeff on the line.
Tony
We can have Brandon make that call.
Speaker
Can you give a feel for what kind of connection we need for the amount of data changes that you have from day to day? Let’s say your whole organization changes by 10 gig, is it reasonable to transfer those block changes?
Ken
Absolutely, it is a pay-as-you-consume data storage, on a monthly basis. Are you talking about is it reasonable for us to accept it or is it reasonable passing it? Getting it to us?
(Can’t hear the other speaker)
The bandwidth really between your location and the Internet, because we’ve got four meg bursts to 20 coming into our location, so in most cases it requires folks to take a look at exactly what they are doing on a daily basis to ensure that the changes get to the Internet. If you’re doing 10 gigs a day, I’m sure there is a calculation we could do to make sure it is optimized or not optimized. I guess you would do a T3 if you have fiber to your building, I don’t know if you guys have fiber or Metro E or something along those lines.
Speaker
I was going to comment on other connectivity options. Right now we two T1s that are bonded together from our Internet provider giving us 3 megabyte which is going to be a lot cheaper than T3 I think. But a Metro Ethernet can get you a 10 megabyte connection and the prices we’ve looked at for that are about the cost of 3 T1s.
Ken
And those folks are cutting each other to ribbons in terms of price.
Tony
I’ve just in the past couple of days gotten AT&T pricing on 10 megabyte fiber and it is about $2,300 a month, and other guys are cheaper, no doubt about it, and maybe unreliable, maybe more reliable, depends on who you ask or which day of the week it is.
Ken
And it depends on how much other traffic, like do you have somebody in your office listening to somebody over the Internet, the radio. It might even be cheaper or smarter to have a dedicated, maybe as small as a T1 just to your data infrastructure so you can pass the block level changes to us, cause that’s what we do, we basically take the block by the changes into our environment on a continuous basis. What we ask is that you guys check or set up some kind of alarm to ensure that it is leaving your location, I’m sure you could set up some kind of a something that would say “hey we have a problem communicating” but the only things we run into that I’ve seen problems is size of bandwidth or size of pipe out of there and is that particular pipe up and running and happy and receiving and passing information.
Tony
I would imagine that on your data center, you can effectively manage that we are sending you data, that that would be just like a managed data circuit.
Ken
Sure, and I’ll also ask Jeff, and I know that we want to provide some kind of view image too so you can see off of a management console your particular walled-off environment.
Let’s try to get Jeff on the phone.
(Dial tone, then static, then Jeff’s voice mail…trying to get Jeff…background talk)
Tony
Simon, you get to go first.
Ken
I’ll run through the five that I have right here, then people can ask you questions. (have Jeff on the phone now).
The first question Jeff is we talked about how we back-up the back-ups and then the offsite, can you expand on that a bit, the methodology, where is the back-up to the back-up site, etc?
Jeff
So we’re talking about VSI coming to us and how we get it from our facility to our back-up facility?
Ken
Let’s say we have gone into our customer’s location, we’ve virtualized business critical servers, done the data extraction, we’ve put it into our location, we are receiving block level changes and now it comes time for us to do our back-up and offsite.
Jeff
We use a combination of two different technologies, because everything is virtual on our end, we take a VCD, or a VM ware consolidated back-up snapshot of all the systems, we do a disk to disk right off of our san to external storage arrays, and we take that to an offsite data protection facility, like an iron mountain and rotate those out on a weekly basis, that’s one level of protection that we use. The other is the VPN side of the house which is more of a granular, file level database snapshot back-up facility where we can actually provide recovery service for all file level SQL and Exchange databases to roll-back to particular points ib time without needing to go to another sort of facility like a tape or another disk to disk type back-up, we can pull that data back for recovery purposes, we maintain that on our storage area network and keep all the versions of all the files and databases online so if you ever needed to restore back to a particular point in time, we could do that very rapidly.
Tony
You said you keep all the versions, how far back does that go?
Jeff
It all depends on the customer’s request, you give me the number, we can allocate the amount of storage required to do that, typically the default setting is a 14-day roll back, if you need 30, 60, 90, 120, 365 whatever beyond that, that can be provided as well, but the default is 14.
Tony
So it’s just a matter of paying for more storage?
Jeff
Essentially, yes, because obviously, the VPN back-ups have to allocate the amount of space required to keep those snaps. So yeah, it would be representative of the amount of storage consumed total, and because of the amount of storage that is consumed, we would keep track of that based on what is actually consumed rather than what’s allocated because of the nature of our storage environment, we can very accurately represent how much storage is being consumed.
Tony
Ok, that makes sense and just for discussion here with everybody, the assumption is, and I’ve seen this from other vendors, they say most of your back-up everytime, you are backing up the same stuff over and over, you go to your 10-year old back-up and your last week’s back-up and 70 or 80 or 90% of it is the same stuff, so in this case we’d only be paying that extra 10% to keep that additional stuff for that long period of time.
Jeff
Exactly right, the way VPN works is after the first initial synchronization, establishing a good baseline of all the data that is out there, only the block-level changes that have occurred to those particular files or databases or what have you are actually synchronized, so the amount of space beyond that then is incredibly optimized to only track those block level changes as you go forward in time or as the amount of time that we are allocating for the roll-back, if it’s 14 days or 30 days, obviously the longer out you go, the more efficiency [Time Stamp00:39:47] you are going to have because you are not taking a copy every single day and storing it somewhere, you are just taking one initial copy and then pulling just the changes down from day to day or hour to hour, how ever often you want the snapshot to be.
Tony
Great answer. Now talk to me just a little bit about how that granularity works with SQL databases, with Exchange mailboxes and all these things that are huge massive gunk and I want to restore a piece of it, do you have a tool that helps us there?
Jeff
Yes, the same theory applies to SQL databases and Exchange databases, because what the VPN system does is it actually tracks it at the transaction level, so if we need to restore, let’s say we are taking snapshots every four hours, what that does, the system is going to put a marker designating which transaction files have actually been pulled for that particular snap-shot period, of course with the back-up of the database and the transaction log, VPN can then recreate what the database is at that particular point in time. And that goes for SQL and Exchange both because they both work essentially the same way, they have a main database file with transaction logs and you can get to any point in time you want by taking the base database information and then rolling into the appropriate transaction logs to recover that point in time snap-shot of that database. Now of course with SQL, it’s databases made up of tables, from the SQL perspective, you can restore particular databases back to a point in time, with Exchange they go just a little bit further down the chain, where you can restore storage groups, entire databases, or at the mailbox level as well. They don’t as of right now have the ability to restore individual mail messages but they will restore mailboxes themselves, very similar to the [Time Stamp00:41:49] back-up exec brick level [MS?] recovery.
Ken
Jeff there was question regarding requirements on the client’s side for the server infrastructure.
Jeff
For which piece, the DR piece?
Ken
Yes, if they have certain types of servers, the question was, is there anything that I have that could be virtualized or?
Speaker
Different companies here have different technologies and how they use VM ware, some of them don’t, some of them have a san, some just use local storage, I was just wondering what pieces are needed in our network, do we have to have stuff virtualized or can it just be done with plain old traditional servers?
Jeff
Are we talking about DR back-up, or virtualization VSI, which aspect?
Speaker
I guess I was thinking more about the DR stuff but how would it apply to all those?
Jeff
Let’s say we are going to do block level replication, it doesn’t matter if it’s a physical box, it doesn’t matter if it’s a VM, it doesn’t matter if it is Microsoft virtual server as long as it’s operating system, physically in this case a Windows operating system if we’re doing [Time Stamp00:43:33] block level or file level replication, if we’re going to be doing replication at the san level, there are other utilities that we use or at the low low block level of the disk level to replicate a Windows based system, but in either case, it is just an agent piece that sits on there and actually intercepts all the [MS?] that happens on the disk and then gets that ready to be transferred over wire to our data center to where we write it out as a virtual machine, so at the end of the day the resources that you send us do get virtualized one way or the other whether you are sending it to us as a VM already or if it is a physical machine, it makes no difference. I think that’s what you’re asking as far as which technologies are available for the different systems or platforms that you have. Now if you have non-Windows based systems there is also another method we use, this ties in with Data Core technologies using the ING technology. The ING technology if you are already running Data Core makes no difference what the operating system is at that point, you aim the data to us in our facility and we write it out the same way except that now the ING piece is actually sitting below the operating system so it’s what they would consider operating system agnostics, it doesn’t care what it is, it will take that as the block data that it is, the ones and zeros and pull it across the wire the same way to our facilities and we store that in the same exact way we do all the other data that we have for all our customers. So there are two different levels at which we look at this, one is more if your facility is mainly Windows based, then the block level file level replication at that level is probably going to be a little more feasible for you, but if you’ve got a mix of different operating systems, then we would need to look at placing some sort [Time Stamp00:45:45] of san melody san symphony implementation for your entire environment to run on and then we could replicate all that regardless of your operating system. Hopefully that answered your question.
Ken
Jeff, can you touch on what we do on a month end month out basis and whatever the frequency is testing the integrity of our infrastructure? Like our own disaster recovery type testing.
Jeff
Sure, most of our environment is made up of either DPN protection or VCD protection, we will on a regular basis map those image files the VCD creates for us, which essentially is just the raw disk files that we pull off of our san, if you’re familiar with how VCD works, it literally just goes in and scoops up the entire VM representation the file that represent the VM and puts that on our disk to disk back-up architecture and we move that offsite, well when we do a testing of course we will actually take those same files and use VM ware converter to mount those files and do test restore to make sure we have integrity on those files. Of course there is also integrity done at the time of back-up as well. VCD will not allow a back-up to complete against a particular system if there is any corruption detected in the file stream, because again VCD works at the fiber level over our san so it applies the same checks and balances that it does to any level just because we are going in and scooping it up while the system is running, because of the way VCD works, it allows us to do that, the same things apply, everything associated with moving the file from location to location is already in place, then the VM ware converters will validate that file to check the integrity of that file to make sure everything is in tact. As far as our VPN data is concerned, our VPN system is constantly going through and checking the integrity [Time Stamp00:48:12] of those files and base snap-shots it’s taking against all the data we’ve pointed it to, so that’s something we get an ongoing report from the system as to whether or not our synchronization needs to take place because either the synchronization didn’t complete or their was a defect it found in the data set for the course of the number of days we’ve been protecting the data, that’s an ongoing thing, a little bit more of an automated task where the other one is more of a maintenance thing we do.
Ken
One of the participants asked me where the physical location is, which we know is Boca Raton. I’ve not seen the host.net facility, can you describe the actual facility and the ‘what-if’ of if there is a hurricane bearing down on Boca Raton, what are we gonna do now?
Jeff
The facility that we reside our whole entire infrastructure on for not only our own stuff of course but as well as our customers is a building in Boca Raton where they’ve taken the building codes and doubled the building codes, so the roof, for instance, is rated at 245 mph winds which I don’t think we’ve seen in Florida ever and the infrastructure itself, all the concrete walls, the roof underneath the roof, concrete upon concrete upon concrete multiple levels, category 5 rated facility, all the glass is not only high-impact glass from a hurricane perspective but it is also bullet-proof glass so everything is very well secure from the ground floor level. All the data center side is entirely above ground, it’s above the first floor, actually a second-floor building and so that’s why they put a lot of attention on the roofing as well as on any type of windows that would expose it to the outside, and of course, being that it is already 14 feet above sea level and it is on a very steep gradient, any water that would accumulate [Time Stamp00:50:48] anyway would run off down the hill to the main road area, so you’ve got 14 feet above sea level, the additional nine feet for the first floor, from a flooding perspective, flooding is 23 feet, we’ve probably got some other issues to look at, but that has never happened and the drain systems have been designed to handle any sort of flooding that would happen in that area. So flooding is a very far out possibility, not gonna happen anytime soon. As far as the rest of the infrastructure is concerned, the building obviously is a new building, they have 1.5 megawatt generating capacity that they also supply the fuel for, the companies that supply the fuel also happen to be hosted in this facility as well and the reason for that is they also house all the emergency services for all the surrounding counties in the data center. So they have contracts saying that we are guaranteed that we are always gonna have fuel besides the 12-15,000 gallons they have on site, with the 3 generators they use, they also a contract to guarantee for those emergency services at that data center always remains powered on. As far as carriers, they have all the major carriers coming in, AT&T, Bellsouth, everybody comes to that one point for redundant circuitry so of course we have taken advantage of that, not only is our entire infrastructure redundant but all of our connections to the Internet are redundant and they all back to each other and load balance across each other automatically. So as far as facility is concerned, it is probably one of the more rugged built facilities that I’ve experienced in south Florida, and they’ve had a good track record history with maintaining as close to 100% possible up-time for all the systems and of course we’ve taken it one step further, most managed data centers don’t allow typically the use of UPSs in their racks if anyone has ever been in a [Time Stamp00:53:12] managed facility, they like to manage that for you, they supply the power, they have their own generators, their own batteries, everything is entirely managed from that perspective, but we kinda said well, the amount of data and a critical nature of the data we hold is one that cannot sustain a microsecond interruption in power so we have aa3 main pieces of power coming into our facility, one is off of the host.net battery and generator side, we have another one that comes from them outside of that realm for direct power into our infrastructure outside of the control of host.net and then we have a direct feed coming in outside of the host.net altogether from FPNL which we feed all our own UPSs with, so even if there was an issue with power in the building for whatever reason, if that got disrupted in any way, we would always remain up because we have our own UPSs running the environment and that is something we said we have to have or we cannot put our infrastructure in this facility. We have to have all avenues of power covered because there have been times in the past where hosted facilities are supposed to be 100% rock solid up always, but as you know, things happen, so we’ve taken every precaution we possibly can with regard to power and regard to Internet connectivity for all of our customers. Honestly, we really couldn’t do any more than we are doing now for the data we are protecting.
Ken
We had to fight with the host.net folks to get them to allow us to put our own alternate power supply in, they were not too happy with that, but we went to the wall and were able to put our own as well.
Tony
I guess after 365 Main’s problem, that was an easier discussion to have.
Jeff
There have been times when it shouldn’t have happened but it did, and we brought that up and said we can’t do it, we protect on the order of 40 [Time Stamp00:55:34] to 50 terabytes at any one time and we just cannot have an interruption of power, period, ever. The systems are turned on, they never get turned off. That’s it. They knew when we came in to install, they knew we knew what we were talking about and they said fine, we’ll make this one exception.
Tony
Sounds great. Even though this is outside your control, can you talk to us about how the connectivity comes into that building because I figure the building is there, it withstands the hurricane, but there is two feet of water everywhere and all the fiber services are flooded, how do you maintain the connectivity through that, or how do the other people maintain connectivity, what’s the trust factor there?
Jeff
Well, I couldn’t talk to the actual infrastructure surrounding the connectivity that they use, that would be a Jeff David question, he is the president of host.net, he would be able to answer that, but what I can tell you is that all the major carriers trunks converge on this one point and there are multiples of them, so besides the fail-over that we have for the two carrier lines that we’ve got, there is also additional fail-over that they have outside beyond our switching and routing that we’ve got in their own facility that will also handle, if it is a sonnet ring fail over or metro e fail over [MS?] or whatever the case for whatever carrier they happen to be failing over to, that happens automatically outside of my control then further guarantees, if you think about it, the building is standing, the power is on, but if the Internet down, what’s the point of any of it happening at that point, so if you don’t have connectivity, power’s on, great, but we can’t get to anything. So they’ve taken a lot of steps which Jeff could speak to because he built the facility, surrounding the amount of redundancy they’ve put beyond my control [Time Stamp00:57:56] at the network layer, but I can tell you all the major carriers come in there and terminate there to give us the maximum up-time as far as network connectivity that we could possibly have.
Speaker
About what is your current capacity, how many servers are you actually running for customers and what do you normally run on the average day, and if there is a large problem with a large ISP and let’s say 20% of your customers had to fail over their servers to you, what’s your availability to handle that?
Jeff
First question, how much capacity do we have on any given day? The majority of our systems run less than 15% across the board, and that’s for all the hosting services that we have, we’ve never seen it go higher than 15%, all the total systems right now are on the order of 70 to 80 different systems that are always running with probably another 30-40 systems on standby, the capacity to be able to handle the load of a particular fail over would certainly be easily met with the capacities we’ve built into the systems already. If all the Atlanta area went down and we had to take over responsibility of running those particular systems, what was the question?
Ken
Let’s say somebody with a big backhoe hits one of AT&T’s main lines and so we have 10 customers on our disaster recovery business continuity, so now they are completely blind and the phone starts ringing, what’s our capacity to be able to handle a spike, if you will of multiple customers needing us right away?
Jeff,
The only thing that would have to be addressed which we know could happen quickly with host.net, if we needed to ramp our circuits as high as 100 or 150 megabytes per second, that is nothing more than a phone call. As far as from a systems load perspective, that would never be an issue, we built in enough capacity to be able to handle [Time Stamp01:01:32] the demands for the actual VSI, the systems running inside of our infrastructure, the only question would be calling up host.net and having them ramp those circuits up to the amount that we would need for that time and that can be done in a matter of minutes, so we could respond to it very quickly.
Tony
If you have to be able to fail over, obviously means DNS changes very quickly, is there an automatic assumption that you guys take over our DNS for us?
Jeff
Yeah, that would be part of a logistical lay out that we would work with each customer by customer basis, each customer has a totally different set of requirements from company to company, but yes, if you have a set of systems that need to have Internet visibility all the time, then yes, the DNS changes and whatnot, we would work through the logistics of getting those changed over so that the fail over, once the systems are actually running, everyone will be able to find them on the Internet. There are a couple ways of doing that, but yes, each instance is going to be based on what the customer requirements are. If there is total loss of connectivity then obviously there has to be a total cut over of all the DNS and all the pointer updates, if it is not a loss of Internet connectivity, but just a fail over or even a work load redistribution some customers also request to set up a point to point VPN tunnel to essentially allow those VIS systems that are running in our environment to look as if they are running in their own environment, so there is no addressing conflicts. It’s on a customer by customer requirement basis and everyone is different but we would work through the logistics of handling that for you.
Tony
It just seems like it would be to our advantage to let you take over our DNS and then whatever [Time Stamp01:03:44] happens, you can manage it. So even though you’re normally pointing back to us, the one central point of management, which right now, with us we don’t have the world’s greatest DNS management the way we have to go through our current ISP, so you could actually improve things for us right there, which would be a plus.
Jeff
Absolutely, if we were actually hosting the DNS, which would be far easier than having to deal with the ISP directly, those changes could be made very quickly, if we were the authority for those demands, the address change would be within minutes, a phone call, something happens, start these systems up, change the DNS structure and everybody is hitting the new addresses immediately. And that’s customer by customer, some systems are more critical than others, some have to be Internet facing all the time, some do not, so it is a mixed bag depending on what the customer requires, but that is a possibility as well, hosting of the DNS system altogether to make those changes very rapidly.
Speaker
Along those same lines, how are you staffed? I understand that if we at Perimeter Church call in and say we need this changed, that’s great. If you have a dozen customers calling at once, things along those lines, how are you staffed to handle something that would be on a much more massive level effecting a number of clients?
Jeff
If there is anything along those lines of a fail over that would need to take place, obviously you would have all of our direct numbers as well, but the main number goes out to all of a pool of our engineers that are on call all the time throughout the day, and one of those engineers is going to get that call, and because of the way we have actually engineered the set up, the interfaces for all these different systems, the fail over perspective from our side, what it looks like, is simply [Time Stamp01:05:55] nothing more than bringing on machines and if it needs DNS changes we make those changes, every customer obviously has a document that represents the systems that are running, what they are, and what needs to happen. We don’t like to take a failure and have that be the first run-through. If we are in this scenario, this scenario has already tested, well documented and all the considerations have been put in a document to allow any of the engineers to get that call to be able to say ok these are three main ones that have to come up, these DNS have to change, so on and so forth, and this can happen very quickly because we’ve done all the research and all the documentation prior to an actual disaster happening so that we are not guessing when you call us up and we say we need to make that happen, that gets executed very rapidly because we’ve got all of our ducks in a row prior to that actually happening, so from that perspective, it would be a matter of calling us direct or calling the main support line and submitting that through our phone system to talk to one of our engineers on call.
Speaker
You mentioned the VPN solutions, point to point, if we fail over, what kind of VPN solution would there be on your end so that our users would see it more of an invisible change for them. I know we change DNS but if we run Sisco so we have our VPN solutions, what do you have on your end then?
Jeff
What we offer as a default for every one of our customers, of course they have the option to change this if they choose, we always offer, we use Microsoft ISA 2006 and it allows us to essentially point any of your user records to your domain to be able to authenticate against it to be able to allow people to come in either over layer 2 or over a point to point tunneling portal call, that’s [Time Stamp01:08:18] typically the most widely used at the Windows, people could VPN that way or you could even have site to site IP tunnels set up and our firewall works with all the major firewalls as long as it is abiding by IP tunnel protocol, then we can set up a site to site on demand. So there are two aspects to that that we can establish VP connections in general.
Speaker
How is a fail over detected? One concern I have is, when a fail over is detected that one of our servers goes up on your site, that’s gonna be aimed for something on the outside and really the server is physically still up, we just lost our Internet connection or something and our users are still using it on our side, now we’ve got two sets of data going in two different directions.
Jeff
Upon a failure, if it is a failure that you clearly know about, we’d be notified by you to control the fail over process, we’ll actually bring the systems up, if there is a loss of connectivity, our systems notify us saying we’ve lost connectivity with the remote site for whatever reason, the system will wait for user intervention, aka our administrator to get word either from you or if you’ve already pre-established a set up rules that say, if this site ever goes down, I don’t care what happens, it comes up on our side no matter what without requiring a phone call from you, user intervention, so on and so forth. Our systems are going to detect that we’ve lost connectivity with your site, if it is something that just happened out of the blue, we will notify you as stated in the pre-installation documentation that we use, we will notify you directly and ask if there is a problem and how you want to proceed, if you just want to correct the problem, re-establish the link and continue or do you want to fail over and then trouble-shoot the problem to find out where the issue is? That is entirely up to you on how we handle that, we can have pre-designated systems so that no matter what happens, if there is a failure it comes up on our side or we notify you of the issue and we take it from there.
Ken
That might have done it Jeff, but I’m sure as soon as we hang up, somebody will have a question.
Jeff
No problem, you have my number.
Ken
Anything else you want to add Jeff?
Jeff
I can’t think of anything at this moment. Just a question for anyone in general to get an idea, if you want to take a show of hands, how many people are running some sort of virtualization today?
Ken
Half. There are 12 folks in the room, good show of hands.
Jeff
Ok, so some sort of virtualization whether it be san or VM ware or Microsoft Virtual, what percentage of all those systems that are running VM ware is Windows vs Lenox vs something else, what’s the distribution look like? Physical or virtual either one?
Ken
90/10 Windows/Lenox. Tony is 100% Windows. It appears that the majority is Windows.
Jeff
That’s predominately the case we are finding as well, I just wanted to point out that the solution we use to do the block level file level replication sometimes in the 3rd quarter this year or possibly the 4th quarter, I believe we will be adding the capability to do the same block level file level replication [Time Stamp01:14:02] at the Lenox level as well. The reason why Lenox is a little bit harder to adopt into this particular solution is because of so many different flavors of Lenox, so many different vendors, different builds of Lenox kernels out there that it’s difficult for software developers to certify on this and certify and this and ya know with Windows, you have a couple different versions but for the most part it is based on the same general code base, where Lenox has a lot of tweaks and changes across all the different varieties of Lenox kernel level stuff, and because what we do sits at the kernel level, that’s very important to know what kernel is there and if you took a list of all the different Lenox operating systems with all the different kernel variations, you could fill a small room with the paper print-out. It’s good for the open source market but it makes it difficult to do some low level stuff like we are talking about here, but as I’m hearing, that will be available by the end of the year and we’ll be able to take in those systems. Unless of course you are already running a san melody or san symphony today, at that point it doesn’t matter what operating system, every operating system on the planet, makes no difference at all, ING technology doesn’t care what the data is, all it sees is ones and zeros. So a couple different ways to approach it. And I think that’s all I’ve got.
Ken
Before you leave Jeff, we essentially sell two services off of this platform, the one we talked about primarily this afternoon was the disaster recovery business continuity services, we also have a hosted virtual services infrastructure which is essentially like renting a house, we virtualize your services, put them in our location and then whatever you put on them, you administer [Time Stamp01:16:24] them, you can use our storage as well, rather than going out and spending 10, 20, 30 thousand dollars on equipment refresh or if you want to set up maybe a test or development environment and you don’t want to go out and spend the capital in order to put that kind of infrastructure together or you’ve run your budget out, we can offer the same kind of space on our infrastructure, you do the management of it, we just say here’s the space, do with it what you want, as long as you’re not running a gambling site or something other than that, you are free to do whatever you want off of it and that’s another aspect of the infrastructure that we have. Like I said, we mostly talked about disaster recovery business continuity because that’s really what Tony and I started talking about a year ago, but the other side of that is the virtual services infrastructure, I don’t know if anybody had any questions on that, it is essentially the same product by a different name and a little bit different methodology but if guys had any questions on that while we’ve got Jeff on the line, we might knock those out as well.
Looks like we got it Jeff.
Jeff
Great, well you’ve got my number, if there is anything else, let me know.
Ken
Thanks Jeff.
Jeff
Thanks everybody.
Tony
So does everybody have they need? Do we want Ken to send us a price list? There’s a broad continuum here so let’s figure out what we want to do next.
[Someone speaking, I can’t hear]
Ken
On a general basis there is a slight charge for protective servers, and there is a charge for gigabytes per month. Tony and I have talked about a descending price scale as the gigbytes go to terabytes, I’ve not developed that yet, but it’s just noddling it out over 10-15 minutes of putting it in some form I can get to you. The rubber hits the road when you and your folks get with our technical folks and Jeff [Time Stamp01:19:28] and the folks on his team and figure out your servers and how the storage is constructed, etc. I step out of that.
Speaker
When we met not long ago, we talked about ways to consolidate some of the things we’re doing, to take advantage of pricing structures without bumping into each other’s infrastructure and I don’t know what those options might be or what they might look like. We’ve gone and priced and we were looking for a pipe for something like 3 grand a month or something a couple years ago and that was for a small cage and stuff like that, so what you are presenting is new and different, so isn’t that what you’re asking Tony? How would we look at doing this? We haven’t even considered it so it would probably be a matter of sitting down and saying this is what we think we’d like to do or needs to happen.
Ken
We are open to that, I’ve got a big honkin’ infrastructure I’ve got to pay off, so, I’m interested in talking to you. As long as we can erase this part where Jeff can’t hear it on the record, I’ll do what I can do in order to accommodate, Jeff and James are smart guys, customer oriented, I’m marginally smart but very customer service oriented as well, so let me know what I can do to help.
Tony
And Ken, just to touch on that same thing, in my mind, this solution and the others we’ve been looking at, they are all variations of the same thing, we are pushing data somewhere, optionally bringing up a virtual server, specifically sharing rack space, virtual server space, so, if we were to use Fairway’s solution and never switch over to a virtual server, I might still be interested in the back-up part of it and just knowing that I could switch to a virtual server if I ever needed to. Go ahead [Time Stamp01:22:05] and virtualize my VMDKs, have them there but never fire them up, that would be ideal because hopefully we are never going down here, right Richard? We never fail here!
Ok, everybody, last chance to ask your hard questions.
Ken
Thanks for the opportunity to come and talk with you all. You have my card, I’m always available on this phone or email. Ask hard questions, we’ll get the answer to you, let us know what we need to do to assist you guys in continually running your server and data infrastructure. I appreciate your times. Thanks a lot.
Last July, I exited
Recent Comments