Cloud Computing Contracting
December 20, 2012
Sabrina: ...you talk about, and they hear these terms and they don't fully understand them move forward with it anyway. This is what we know is what is the sophistication level, and what is the basic understanding level of folks who are having input into these decisions day to day. My goal is when you guys [indecipherable 0:15] , or is this you? OK?
When, essentially what we're doing is outsourcing. We are responsibility for our information and lots of other. End of the day, you can sort the response out back here at the Department of the Interior, or the U.S. International [indecipherable 0:32] . The buck is always going to stop there. Right? [indecipherable 0:34] we had earlier on.
When you're sitting there are you going be able to [indecipherable 0:38] for this. No, at the end of the day Interior is going to be on hook, and Interior is going to be dragged up to the Hill, and you're going to have some angry senator, or congressmen, and on the table there'll be personally identifiable information now released onto the Web.
Why are now we having to pay for identity theft monitoring for people, because their payroll information out there, or even worse, what about grantee information, or contract information? You're not going to be able to sit there and say, "Well, it's not our problem. Google was supposed to do it, and they didn't do it."
You are not going to be able to outsource your responsibility for that information. We always have to think about that when we're using any technology, but most importantly cloud technology. OK. What is it? We're going to talk a little bit about the basics if you are already a cloud expert, then I'm sorry.
Usually, when I can see my audience I have people raise their hand and tell me if they're experts or not, and that's when I say I know where I'm going to get my sniping questions from, but since I can't see you all, you can raise your hand in the comfort of your own office. What is cloud computing? I love this quote by Larry Ellison.
He's the real eccentric CEO of Oracle. He says, "The interesting about cloud computing is that we've redefined cloud computing to include everything that we already do. The computer industry is the only industry that's more fashion-driven than women's fashion."
Maybe I'm an idiot, but I have no idea what anyone is talking about. What is it? It's complete [audio skip] , idiocy going to stop. This is what I'm saying. The whole cloud industry failed, it's directive of words and phrases and things like that, and literally, it's made up. It's all market, the cloud is a marketing term.
Think back, we're thinking about mainframes, right? We're thinking about dummy terminals and we're thinking about locking into a database of information from anywhere you sit. It's really just the past as prologue. Cloud computing genuinely is just some made up marketing term.
What is cloud? We have Larry Ellison trying to tell us that the cloud computing is a lot like women's... [audio skip] government's definition of cloud computing. I'm from the government and I'm here to help. Cloud computing is a model for enabling ubiquitous, on demand, blah, blah, blah. If I read this whole definition, you'd all be asleep.
This is the NIST definition of cloud computing. NIST, does anyone out there used to work in NIST, or if you still work in NIST, or if you know people that work at NIST, I'm sure they're lovely people. But NIST is wildly unhelpful when it comes to any of this stuff. They are habitually 18 months behind the curve, and they usually have no idea what they're talking about.
Does this definition help anyone? I don't really think so. Let's move on to see, trying to figure out, really, what the cloud is. This slide is showing us a traditional computing network setup. We have our cloud out there, the router, the firewall, the brick wall that's protecting us from things. We have our server, and then we have our terminals that are attached to it.
This is our traditional computing structure. When you move to the cloud, what's missing? Usually at this point, I ask for audience participation, but one of the things that's missing, that big brick wall. Where is that security that we're responsible for, that we actually have tangible, hands on ability to build? We can make that brick wall as tall or as wide or as thick as we want.
We can make it out of brick. We can make it out of steel. We can make out it whatever we want. Where is it? Cloud providers are going to tell you, "Well, it's in the cloud. Trust us. It's there." One of the things that you're going to hear me talk about later is, "Well, I'm going to trust you about as far as I can throw you unless you can demonstrate to me that that brick wall is actually there and you also let me in the door so that I can audit and test that brick wall."
A lot of times the cloud providers are going to welcome you into their lobby and then stop you right there and say, "Thank you very much for coming by. We've got this." They'll pat you on the head and will send you on your way. Not acceptable. Not acceptable in the federal government, period. We need more assurance than that.
A lot of the stuff that I'm going to talk about today comes from experience. I have reviewed, probably, about half a dozen federal cloud computing contracts and some of the stuff that I see the providers trying to put in the contracts are ridiculous. There are words and phrases in there that mean nothing. There are things that nobody has even thought about and they're not even worth the paper that they're printed on.
It's super important that if you're involved in procurement or if you're involved in reviewing any of these technical things, you need to ask, you need to feel empowered to ask. I will tell you what, there are no stupid questions. Like I said, the whole cloud concept is made up by marketers. What I always love doing is asking them questions and seeing them squirm because they themselves don't know what some of their terms mean.
It's very amusing. Cloud computing, this is our technical image of what cloud computing is. Let's make it even easier. Let's try and really explain the cloud so that my mother could understand it. Here we go. You've got this crystal vase. This is Great Aunt Emma's crystal vase.
It means a lot to you, and you are wanting to be very protective of it. You're moving so you're like, "Oh, Great Aunt Emma's crystal vase. We need to make sure we protect this." We wrap it in bubble paper and then we put it in a big, sturdy box and we put foam peanuts all around it and then we put that in another big, sturdy box. We put "Fragile" stickers all over it. We tell the big movers not to sit on it or crush it or anything because we're very protective of it.
Imagine that Great Aunt Emma's crystal vase is your network. Your network's holding very, very delicate information, payroll information, personally identifiable information, HR information, essentially it's things that you don't want anybody getting access to unless they're authorized. When you package that vase yourself, you know that it is packaged the best and most securely that it can be. But if you let the big oafs who are packaging all your stuff package it, maybe they put that vase in a package with books or in a box with books or big heavy cast iron skillets.
Imagine that they are tasked with packaging it. They don't package it correctly. They don't protect it and an incident happens. But remember, this is an incident that you had absolutely no control over or could mitigate because you outsourced the responsibility for packaging up Great Aunt Emma's crystal vase.
What happens? All of your valuable information is now leaked because you had no security around it, because you outsourced that responsibility. All of those wonderful memories of Great Aunt Emma in the trash because you chose to outsource the responsibility for the vase, right? Same thing with the cloud, remember, you cannot outsource responsibility.
What are we going to do about it? How are we going to try and mitigate any incident? The only way we're going to be able to do that is by insuring that all of the terms and all of the clauses and all of the liability responsibility is in the documents that we [beeps] signed with the contract. We'll get to that.
Now, we know generally what the cloud is. Let's talk about the different kinds of clouds out there, because yes, there are different kinds. Here we have our boardroom again, and we have the guy at the front saying, the company's in trouble, gentlemen, we're running out of cloud names for our act, with stratus, nimbus, cirrus, cumulus and fluffy. Fluffy is personally my favorite one.
Here we go. We've got different types of clouds. You can see the risk factor on the left hand side. Highest risk is a public cloud, and that's the type of cloud that cloud providers are offering up these days. The only other types of cloud [beeps] that I am aware of at this point is the Department of Defense, DISTA in particular, is building a private cloud that they are offering to components of the DOD, which would be a combination of private community cloud, depending on how you're looking at it.
But right now, what most agencies are being offered is a public cloud, right? Microsoft, Google, I think Verizon has a cloud that they're offering. There are a couple other of the big names out there that you'd recognize.
That's it, it's exactly what they're doing. They have server farms in big, unnamed, unmarked warehouses with air conditioning bills that are skyrocketing, keeping all of these servers cool. Your information is being commingled with everybody else's who's buying public cloud storage. Your information, and the mom and pop flower shop's information, and oh, by the way, the guy that's being arrested on RICO charges because he's racketeering and we have some illegal stuff going on.
I remember a few years ago I was talking to my colleague Troy, and I was saying, "Well, what's going to happen when the FBI busts down the door of one of these server farms and says, "Hey, one of your clients is being brought up on federal charges. We need all the information, and they basically just unplug everything from the wall."
Looked at my CIO at the time was like, [beeps] "Oh, that'll never happen. They'll go in, they'll be very surgical about it, they'll be very forensic," blah blah blah, about it, right? BS, right? There's already been, in the last two years, a half a dozen examples of the FBI kicking in the door, with the guns and the smoke bombs and all that, literally unplugging racks and racks and racks of servers from the wall.
What their justification is, "Well, we need all of this information." What's happening is, the good folks, the good citizens who are following all of their rules and stuff like that lose their information just as much as the bad guys do. They are not taking a scalpel in these public cloud environments. They are using a hatchet.
We need to be concerned about that. If our information's being commingled, and something happens to the provider, and we lose our information, what's the remedy there? I bet that's not in any contract anybody negotiated. Right?
Are they going to pay for us to get new cloud service? Are they going to pay for us to buy new hardware? Are they going to pay for the cost of all of the information that we have to recreate? No one thought about that, but it's already happened, and it's scary. We got public cloud. We got the community cloud, which is something that's shared by several organizations, and you could argue that the DOD DISTA Cloud is a community Cloud.
Then you have a Private Cloud, and a Private Cloud is essentially your own agency's intranet. The intranet that we have maybe you have the RSA token to log in. That's essentially a Private Cloud.
Then at the very bottom we have the fourth thing called a Hybrid Cloud, and what I love about Hybrid Cloud is that if you talk to any of the vendors they were like "Oh, we can build you a hybrid cloud. It's great." Blah, blah, blah, blah. "It'll be a private, and community, and public, and everything will be wonderful."
I will tell you what, all of the vendors that I have talked to about Hybrid Cloud they were all like, "Yeah. Marketing came up with that, but it doesn't really exist yet." I'm like, "Wow! That's fascinating guys. What's going on?" They're like, "Well, no one has really refined the cross-communication between the three different cloud platforms enough to ensure that what should be private, and what can be public is public."
Hybrid Cloud is great in theory, but nobody has refined the code yet in order bring it into the world. If you talk to anybody, and they're trying to sell you a Hybrid Cloud you can also tell them that there's a great bridge in Brooklyn that they can buy as well. There you go. Now, you are all experts in the three types of clouds that exist, and the fourth one that people like to make up. Next, so here we go.
We're talking today about the Public Cloud again. This is what I was talking about when you and your information is commingled with everybody else's. We've got Billy Joe and Bobbie Sue. [claps] Anybody? Steve Miller? Right, got it, OK, so here we go.
We've got Bill and Joe and Bob and Sue and they're all living in this apartment building. When you buy into a public cloud you're buying into something called a multi-tenant environment. Because you are a tenant like in an apartment building along with everybody else that is buying space in the server farm.
We've got Bill, Bill is like super into security. He has an apartment on the top floor and he locks his windows all the time. He changes the locks on his door every 90 days and he has an escape hatch to the roof with a helicopter waiting. He never buzzes anybody in through the doors and he never gives his key out. He is super, super, super hyperaware about security.
Then we've got Joe a couple floors down. Joe's also concerned about security but he travels a lot and so his sister has a copy of his key. She comes in and throws parties at his apartment every once in a while. He doesn't buzz people in through the door, and he never gets food delivered, but you know what I mean. He's concerned about security but he's a little bit looser.
Bob is just the bachelor of all bachelors. Never cooks. He's always ordering out food. Whenever the buzzer rings on the door he doesn't even ask who it is. He just lets people in. He's throwing parties all the time and lots of ladies have keys to his abode. He's just the bachelor of all bachelors.
Sue, Sue is like a super hippie. Her window is open all the time. She never uses air conditioning. She hangs her laundry out on the trees. She's got people backpacking through and camping on her couch and stuff like that.
Here we've got our multitenant environment and here's where I usually have class participation. Who's the most secure in the building? We need to ask ourselves, what is the lowest common denominator here? Sue. Is Bill any more secure? Not really, if Sue's keeping her window open. Maybe he's a little more strict, he's got an extra lock on the door.
But really, when we think about public cloud and multitenant environments, we need to be thinking to ourselves, we are sharing a security environment with hundreds of other people. How secure are they? There's a piece of technology that runs in these multitenant environments and it's called a hypervisor. I'm going to get technical right now for you geeks on the phone.
Hypervisor is basically a superior operating system that is there to [beeps] oversee all of the other operating systems that individual folks in a multitenant environment are using. What was the thing called in "Tron"? The movie "Tron," the main oversight guy in "Tron"? Right? That's essentially a hypervisor.
Well, everybody was saying, "Oh, if the hypervisor is secure, if the hypervisor is secure, everything else is secure. No one will ever be able to crack the hypervisor, it's fine." Again, I'm calling shenanigans here, right? Sony Playstation, their hypervisor was cracked a year ago and a lot of the information, if you heard about the Playstation hack, all that information that was taken out, the credit card information, and the personal identifiable information was taken out through a hypervisor attack.
You may say that in a multitenant environment there's firewalls between everyone and if somebody is really lax with security, you're still going to be OK. But that's not necessarily the case. If somebody can get in through Sue's window, they can get into Bill's apartment. We need to be thinking about the risks that we're willing to accept in a public cloud multitenant environment.
Now, on top of the different types of clouds that I went over, public, community, private, and hybrid, there's different service models that you can buy within each one of those types. We have applications, platform, and infrastructure. When you think about this, think about it this way. Do we have a question?
Amile: [indecipherable17:12] master control.
Sabrina: Oh, master control. Thank you. Thank you, Amile, you're the best. Master control. Love me some "Tron," thumbs up. When you're talking about service models, think about this. Think about when you go and you open your computer in the morning and you go to open a Word document. You are the client.
Word is the application. Word sits on Microsoft Office, which is the platform. The infrastructure is the Windows environment that your CIO's office has deployed on the server. Your information sits on the server.
When you double click that little Word icon, you're going through application, platform, infrastructure, sever and back in the blink of an eye. You've gone through all of those levels [feedback noise] to present to you a Word document. With cloud computing, you can slice and dice and you can either buy applications, which is considered software as a service, platform as a service, or infrastructure as a service.
For example, Google Mail, right, software as a service. I guess you guys are now using Google Mail. You guys have purchased software as a service. That is the application. Platform as a service is a type of cloud that, I think again, the marketers thought this would be a great idea but they're realizing that there's not a huge market here.
Platform as a service would essentially be my office, we have a super crappy time and attendance application that was built in the 1970s and it uses DOS and there's little turtles running around. It's a total mess. Let's say, for example, in our CIO's infinite wisdom they wanted to move that application whole cloth into the cloud because they think it's so great.
They would use "platform as a service." They would deploy our own time and attendance system put up onto the cloud. They would have to make sure that there's some adjustments that are made so that it would talk with everything, but we would use platform as a service at that point.
Infrastructure as a service is actually getting a lot more traction as well. The two areas of cloud computing that are really moving the most are software as a service and infrastructure as a service. Infrastructure as a service allows for rapid deployment of basically memory, of space.
An example that a lot of the cloud salespeople like to use is "cash for clunkers." The cash for clunkers program was actually super, super successful. People loved it, and the Department of Transportation ended up getting a lot more applications than they thought they would. They got so many applications within the first about 48 hours that the website crashed.
People were crashing it because they were trying to get to it and apply. One of the arguments that the cloud providers will make as well had Transportation deployed this on the cloud, we could have quickly expanded to provide for more space for the influx of folks that were trying to use the system. Yeah, arguably that's true.
A lot of folks use infrastructure as a service, too, if they're designing applications or testing a lot of applications or they're using things that are very information intense. When you need a lot more space, but you only need it for a short period of time, infrastructure as a service can be a good idea. Within our different types of clouds, our public, private, community and hybrid, we can have different types of service, software, platform, and infrastructure.
Kind of like a Chinese menu. You can slice and dice and you can pick out whatever you want, but it's important that you know what you want before you get there. Here we go. We've got Alfred E. Neuman. What are we worrying about with cloud computing? Well, we're going to worry about a lot.
How many of you guys have seen the Gartner Hype Cycle before? Well, I can't see your hands anyway. You can raise your hands in your offices. I love this. I think this is great. The hype cycle says you've got a technology trigger, you shoot up really fast to the peak of inflated expectations. Those are the marketers and the salespeople all the way.
When you buy the thing, then you finally realize that it doesn't do what you think it's going to do, and you hit the trough of disillusionment. Then you realize that you're stuck with it. You're like, "All right. What can this thing do?" You tinker around with a little bit and you realize that you go up the slope of enlightenment. It can do some stuff, and you hit your plateau of productivity.
Now Gartner has updated their hype cycle for 2012, and I think it's great. You can see along this hype cycle where things are. You can see community cloud, cloud security and risk standards, still very low on that peak...the upward trajectory there. The general cloud computing and public cloud storage has started making their way down to the trough of disillusionment.
Cloud stuff is still way on the left side of the hype cycle which we need to be concerned about as consumers. We cannot believe the hype. Don't believe the hype. We need to make sure that we're looking into these things and we're not getting rolled by some of the sales folks and the marketing folks. When you are considering moving to the cloud, there are really only two questions I want you to ask yourself.
Number one, is there a business case for moving into the cloud? This is a simple question. The follow up question, though, and I don't have it on here is, "Is the business case not written in crayon?" I have seen a number of business cases written by CIOs and senior managers and they make no sense. Essentially, they're like, "Ooh, the cloud is cool. I want to go to it. It will save us money. End of story, next slide."
That's not going to work for us. We need to say, "Is the business case sufficient?" When I say sufficient, I mean has the business need been identified not just because it's cool. Has a cost/benefit analysis really been done and have all the risks and costs to mitigate those risks been addressed?
That third one there is near and dear to my heart because I'm a lawyer. I'm all about risk mitigation. Here's eight risks that I've identified that you want to address up front. Now, all of you should have received the document, "Questions to Ask." It's about an eight page document. Essentially, it goes through all eight of these pieces in depth.
It's about eight pages, right? Six pages, maybe? It goes in depth, six pages, on each one of these. When I talk about these things today, I'm not going to go super in depth with the questions, but you all have this document. You can do whatever you want with this document.
When I talk to the inspector general community, I typically tell them that they can do pre-procurement audits or post-procurement audits. But if you're in the procurement side of the house, or the technical side of the house, take these questions to the meetings with the vendors and ask them. Make them give you real answers.
Don't let them pat you on the head and send you on the way and have them just say, "Trust me, we got this," because that's never the case. Our first issue, and everyone always brings up data security and access, data security, data security, everything is great, it's fine, it's all secure. What I usually say to them, if I have a very, very difficult vendor, is I say, "Fine. Let's take security off the table for now. Let's talk about these other seven things. If you can get through these other seven things, I'm happy to talk to you about security."
But everyone wants to start with security, so we'll start there. I love the little cartoon we've got, the CIO guy talking to the person, the employee thing. "I know it's [beep] [indecipherable 24:51] out, I didn't need to battle my way into work through snow. I could have worked from home, but I left all my login details in my desk drawer."
This demonstrates that the lowest common denominator is always the carbon based life form in front of the computer. That's always our main security risk. We need to ask ourselves these other things. We need to say, who has access to live and backup data? Who has access to network traffic? Are we going to be able to run Einstein? How's tick going to impact us? Incident response? Super, super important here.
I love that incident response is this. When you think about a cloud contract, you think about any contract. You ask yourself, well, how many parties are there to a contract? Typically, people say, "Oh, there's two parties to a contract, the contractor and the person who is getting the service or the good."
Well, with cloud computing, there's actually three parties to every contract. You have the main cloud provider, which is going to be the big guy, the Google, the Microsoft, the Verizon, the whoever. Then you're going to have somebody in the middle called an integrator. The integrator is typically a small business, a woman set aside, an Alaska native corporation, whatever, whoever GSA has decided to bless, to provide cloud services. Then you have you, the consumer.
With incident response, it's very interesting. You need to understand what the contract between the integrator and the main provider says about incident response. Then you need to compare that with your contract.
For example, let's say you sit across the table from the integrator and you negotiate hard. You're like, "Hey, I'm going to be responsible for this, so I want to know within 24 hours whenever there's an incident. You have got to tell me within 24 hours so I know and I'm aware." You negotiate hard for this.
The integrator says, "OK, OK, OK. We'll let you know within 24 hours." But if the integrator's contract with the main provider says that the main provider doesn't need to tell the integrator that there was an incident until two weeks later, you are going to be the lucky recipient of information within two weeks and additional 24 hours.
Isn't that great? Isn't that exactly what you need? Stale information. You want to talk about walking into the lobby with all the reporters, you are going to have that as a reality. You need to understand not only what your contract says, but you need to know what their contract says, too.
In addition, you want to make sure that everybody's on the same page when it comes to the definition of an incident. What is an incident? I will tell you that none of the contracts that I've reviewed define incidents. Boy, oh boy, it's like nailing Jello to the wall, right?
Really, anybody can come up with what an incident is? Oh, my information is now publicly available on Google, but that's not an incident? Sure as heck not, especially if Google negotiated the contract and nothing's in there. You need to know what these documents are all saying.
We talk about physical security, vendor obligations and disposal of data. You need to know, these are just some of the examples of things in this area that you need to be aware of. Now, where'd my mouse go? Oh, there it is.
I talk about GSA a lot. God bless them, bless their little hearts, GSA is trying so hard, but GSA is not at all the expert in this area. [audio skip] ...for a few years, an expert in this area. Up here, I have a screenshot of Apps.gov. Apps.gov no longer exists. Apps.gov was the Vivek Kundra's, big...his whole, "I'm all into cloud computing and the government should go into cloud computing. Now I'm going to leave and go to Harvard."
Actually, now he's at Salesforce. Isn't that convenient how that works out? The federal, the first federal CIO comes, tells us all we need to do it, doesn't stick around long enough to see what happens, and now he's trying to sell us Salesforce. This is a screenshot of Apps.gov, when they were trying to push cloud services.
I love this, this is buried in one of the pages, this yellow box. This yellow box says, "Before using or purchasing the products and services on Apps.gov, please do so in accordance with your agency's policies and procedures pertaining to..." and it lists all of this stuff, procurement, cyber security, blah blah blah, and any other federal mandates.
What GSA was essentially doing was saying, "Buy this stuff, buy it right now, it's super important, and oh, by the way, we are disclaiming any responsibility for liability." What GSA tried to do, and they didn't do it very successfully, was negotiate contracts that they said would be palatable to the federal community. For example, they were negotiating out the clauses that say, we are going to agree to arbitration under the laws of the state of California.
Well, we can't do that. We're the federal government. They negotiated out advertising and endorsement, well that's fine. That's easy enough to do, but GSA was not negotiating any of the hard stuff and they were presenting these quasi contracts on this website telling everybody to accept them and then telling them, "Oh, by the way, when everything goes to hell in a hand basket you are on the hook."
GSA was doing nobody any favors. Thank God Apps.gov no longer exists. However, we have this other delightful thing called FedRamp, again, on the GSA website. Oh, GSA, how I wish you would go away. Now what this is is a screenshot, as of yesterday...Do any of you know about FedRamp?
It's a federal...What is it? The risk and something something management, whatever. FedRAMP's been around since 2008. They have decided to finally start doing some work and they started doing some stuff this year, at the beginning of this year.
But as we can see, FedRAMP has not authorized any cloud service providers. No cloud service providers have formally met FedRAMP requirements or have been granted a FedRAMP provisional authorization. What this is telling me is that no federal agency should be in the cloud yet. As we all know, people are.
This is frightening to me. This is frightening to me, because the agency, although I don't know how really qualified they are, but the agency that we have all deemed to be qualified enough to determine that certain cloud providers have met requirements are saying, nobody has. If the vendors are telling you that they have, or if they're telling you that they have met FedRAMP light, which I have also heard, which is super amusing to me, you can go back to this page and say, "Oh, by the way, I don't think so."
Again, we need to know exactly what these cloud service providers are promising us and how they're promising us this information. My parting fact, my parting thought with this is, don't trust GSA. Don't trust them with anything. I've only had awful experiences with them. Do your own homework and let GSA do whatever it is they think they're doing.
All right, so our second area of concern is compliance. We're the federal God of compliance stuff that we need to think about. Privacy, discovery, 508, accessibility, all that stuff. Statutorily required compliance. How are the providers going to comply with that?
Also, if you are responsible for law enforcement or have intelligence information, this is what I love. Do this sometime. You're sitting across the table from a provider, and say, "Oh, this is all well and good. How do you guys handle a classified spill?"
Silence, it's like tumbleweeds going through the conference room. Because nobody has thought about how they handle a classified spill. I got information from a security department from an agency that will go unnamed, about exactly that thing happening. A very ambitious, younger employee had a classified document and attached it, somehow got it from the classified network into the unclassed network and attached it to an email cloud system.
OK, so remember, we don't even have responsibility for the servers anymore and the cloud system was in a public multitenant system. How on Earth do we clean up after that? What I was told that the security office was told was that the person admitted it fast enough that the provider could go in and clean it up, but they were never given any written documentation that that, in fact, happened.
I love asking them, "How do you handle a classified spill?" It's a great thing to see deer in headlights. The other thing is you're arguing with law enforcement right now, the FBI has actually come out and said that there are no cloud service providers that are criminal justice information system compliant, CJIS. If you followed what was going on with the Los Angeles Police Department when the LA city government moved to the cloud and the police department dug in their heels and said, "Oh, heck no."
Google had promised them at first that everything was going to be fine. Everything's great. Move to our system, you'll be fine. You're just scared because it's new and scary, and we'll hold your hand. Well FBI came out and said, "Uh-uh. Google is not CJIS compliant," and LA, the LA Police Department which works a lot with the FBI said, "We can't move. If we move to the cloud, we will not be able to do our work."
What ended up happening was Google is now paying the City of Los Angeles to upkeep their legacy system for the police department. They didn't market that very wide and clear, but that's exactly what's happening in LA. We need to be thinking about the other types of systems that our system is going to be talking to and ensuring that it meets those requirements.
How are we going to handle a non-disclosure agreement? Are we going to ask the provider to have all of their employees sign non-disclosure agreements? How are we going to ensure that people have the proper background checks?
I had one agency that tried to do that. The providers laughed and was like, "Are you serious?" and didn't even talk to them about that. How are we going to respond to things in a timely manner? For example, FOIA has time requirements, you have to respond to things within a certain period of time. If the vendor doesn't comply with that, then we're on the hook for not meeting our time requirements.
We've lots of things to think about in the compliance range. In the third area, we've got termination and transition. It's like when you're dating somebody and everything is great and puppy dogs and ice cream in the beginning and you love each other and everything's wonderful. Then you realize that he leaves his socks on the floor, or she picks her teeth at the table, and the relationship goes down the tubes.
Everything's great when you're negotiating in the beginning, and when you break up, everything has gone to hell. We need to think about, when we're terminating an agreement or we want to transition to another cloud provider, how is our data going to be protected? How's it going to be conveyed to us and how's it's going to be destroyed? How are they going to prove that they have, in fact, done that?
I love this story. The Peace Corps IG was telling a story of before she went to Peace Corps, she worked in the private sector. The company was moving to a cloud-like system. They had a young attorney, she goes, this attorney was very ambitious, loved technology, was very excited. It wasn't me, by the way.
They had her negotiating this contract. They got to the termination and transition portion of the contract and they asked her, "In what format would you like your information when your contract terminates?" She goes, "Oh, what format?" She's thinking and she's thinking. She's like, "Hmm. Formats. What formats do I know? .PDF. .PDF sounds like a great idea.
"Yeah, I want all the information conveyed to me under the end of the contract in .PDF format." Wow, that's the worst idea ever, especially when you have terabytes and terabytes and terabytes of information. This contract happened, it was two and a half, three years. [beep] At the end of the contract, they were done and the company was like, "Hey, we're about to give you all your information back in .PDF format. Get ready."
The CIO folks were like, "What? No, no, no, no. Wait. This isn't going to happen." They renegotiated it in a machine readable format and the provider was very happy to renegotiate but it cost them more money because it was a contract modification and in order to do that it was going to cost them more money.
This has become a theme that we're finding among folks who have jumped the cloud environment early. There's a lot of things that have not been well thought through. The vendors are never going to say no. They're always going to be like, "Oh, yeah, sure, we can do that. But it's going to cost you more money."
The Department of Agriculture ran into that. They moved their information into the cloud. They ended up getting hit with mad cow lawsuits. Everyone remembers mad cow, no one was eating hamburgers for a while. They got sued because of the mad cow stuff and they got an e-discovery request.
The e-discovery request asked for some archival information. The archival information had been transferred to the cloud and was not in a usable format. It had to be pulled out and reformatted in order to be read by a human.
That was not contemplated in the contract, and so it ended up costing them almost a million dollars to comply with one e-discovery request. That was not budgeted in, because nobody had thought about it. But you can't not comply with an e-discovery litigation request.
That really sucked. But it was a learning experience for them that it was things that they had not thought about. Back to termination and transition. What format will the data be provided? What is the timing? How quickly does the provider have to give you the information?
What happens when these providers go bankrupt or are sold or merge with somebody else? Who's, like with Iron Chef, who's cuisine will reign supreme? Well, whose contract clauses will reign supreme here?
What if you have really negotiated hard and you've gotten everything you want and the company merges with someone else. What's going to happen with the relationship that you had with an entity that no longer exists? You need to think about these things.
All right, number four. Asset availability and bandwidth. Now, I know I know this is applicable to you guys because a lot of you guys are all over the country. Do you have people all over the world, too?
Amile: [indecipherable 39:38] .
Sabrina: All right, country a lot of times. Some of you are maybe in rural areas where the connectivity is not so great. What are you going to happen if there's outage time? How frequently is it out when you're in some of these rural areas? What are the bandwidth capabilities?
If everything is in the cloud, and you need a big fat pipe in order to access it. What if you don't have access to a big, fat pipe to begin with? What data availability and disaster recovery is there?
In the spring of this year, there were a bunch of data outages. Google had a couple. Microsoft had a couple. I remember that my IG at the commission was talking to the head of the RAC board, the recovery act board. They were set up super quickly and they set up everything in the cloud. I think it was the Google cloud. When the Google cloud went down, it was down for like four days, I think.
My IG was talking to the guy at the RAC Board and was like, "Oh, how'd you guys fare? Were you able to get through it? What did you do?" He was like, "Oh, we were fine. We were only down for about five minutes, and then we were back up again." My IG was like, "How did that happen?" He goes, "Oh, we have a backup cloud." The RAC Board was paying for two clouds.
One of the sales pitches that the marketing folks will give you about the cloud is that, "It's a redundant redundancy, and nothing will ever take it down, and we're all over the place at once, and we're omnipresent, because we're a cloud." Buh-duh-duh-duh-duh-duh. Not true. Not true at all. There have been plenty of examples where folks have been down for days because of a lightning strike, or because of human error, or whatever it is. Think about what you are buying.
The other thing I like to think about with availability is, let's say, for example, Microsoft pushes out an update, and that update does not play well with your legacy system, and you can't access anything now. If you're the only customer in that multi-tenant environment who's being affected, do you really think they're going to pull that update back so you can access your stuff? No.
They're going to say, "Oh, we're sorry to hear you're having a problem. We're more than happy to send an engineer over at x cost per hour to work with you so that you can get your information back."
Amile: We have 10 minutes.
All right. We're moving. All right, number five and 6. We can get through this. I've been given my 10 minute warning. Maintenance. Again, who's going to be responsible for patching, version control? How are you going to know that this, in fact, has been done? Pricing is a big deal.
A lot of times, folks are going to say, "Well, I'm moving to the cloud to save money." That can be true at times, but what I've found is that it's typically not true, because the out of the box cost that they're going to cite to you is very different than the government cost. What we've got here is what they're going to show you. Hooray! Money's falling from the sky [audio skip] a tiny little bit.
But what you don't realize is, yeah, you may have some savings, but you're paying for the backup for the backup, you're paying for your access to your own data and compliance expenses and you're paying for incident response when somebody doesn't do what they were supposed to do, or they don't even tell you. The costs are going to be there. They may be hidden, they may not be upfront, and what your job as a consumer is going to be is to be finding those costs and making sure that they're upfront.
There's no free lunch. Anything that's free, you want to be a little wary of. You may have some savings up front, but at the end of the day, at least in the government world, our costs are still going to be there. Intellectual property is something that you want to think about. I'm not sure how much you guys at Interior do, but it's something to think about.
If infringing material is found [audio skip] ...these letters saying, "Hey, one of your employees is infringing. You need to take something down." How do you handle that? What kind of indemnity is provided? If there's any claims. Who has access to the vendor intellectual property? Especially for, again, and I give a shout out to my community, the inspector [audio skip]...and audits.
Again, what if Google says, "Well, we can't come and do an audit." We've already had that within the community, providers saying you can't go any further because everything past this is proprietary. That's no good if we're trying to do a security audit. Like, no thank you, we need to know what's going on here.
Finally, again, our IG needs. Please, please, please, I know you guys are the agency side of the house, but don't forget about the IG. We need to have access to stuff in order to do our job, clear and clean access to information and facilities, the ability to do our investigation and access at no additional cost.
What we're finding is that agencies are negotiating these contracts and forgetting about the IG, and then when the IG needs access to do our audits or to do our investigations or inspections, again the vendors are saying, "Well, this wasn't contemplated in your contract but we're willing to let your IG have access but it's going to cost you." It shouldn't be that way.
The IG should have the same type of access that they do now. If the IG is needed to walk to the basement to go check out the servers, it should be the same thing when it's outsourced. Here we go. We've got Dilbert. Ridiculously funny but probably true.
Like I said, we've got his manager, "Let's implement cloud computing so I have something to talk about at the executive meeting. Hooray!" Is this your CIO? Is this your senior management? Is this your secretary? I don't know, it might be. [indecipherable 45:10] it's you. I don't know.
Dilbert says, "Tell them we're evaluating it, that way none of us needs to do any real work." He says, "I like it when you do real work." Dilbert says, "Sorry. I thought you were leading by example."
Is this your CIO or senior management or whatever? "I want to have something to talk about at the executive meeting. I want to throw out the word hybrid and software as a service and shiny things and little blinking lights. I want to sound like I know what I'm doing, and I'm going to take everyone over the cliff with me."
We need to stop the insanity, so we are going to negotiate everything. Please, please, please, I also beg you, find somebody in your solicitor's office that understands this stuff, and make sure that they're coming to the meetings, asking the right questions. I always tell folks that I'm happy to review contracts, I'm happy to consult. I've already consulted with about a half a dozen agencies.
I'm happy to help you guys out if you have questions, but it's really important, since I don't have any authority within the Department of Interior, that you find somebody in your solicitor's office that you can work with. You need to have the right people at the table when you're negotiating these contracts. CIO's office, the client offices, don't forget about the people that are actually going to have to be using these applications.
Your general counsel, information security, you certainly need to have senior management, and please don't forget the Inspector General, but your general counsel is super important, because they are the ones that are going to be looking on it from legal and liability mitigation perspectives. You also want to identify what matters, and this is where we have these funny vague terms that I was talking about earlier. We want to know, who is the contract with?
Is there an integrator, are we dealing with a middleman, or are we actually dealing with the vendor? We need to know if there's three parties to this contract, or just two. Are we agreeing to arbitration? Are there open indemnities that could result in an Anti-Deficiency Act violation? Are they going to be advertising or are they going to be collection our usage information for their own metrics?
Vague terms, and I love this. What does industry standard mean? What commercially reasonable mean? What does promptly notify mean? Again, like I said, this is like nailing Jell-O to the wall, guys. These phrases mean nothing unless they are hammered out. Promptly notified, what that means is 24 hours, 48 hours.
You may be sitting at the table thinking to yourself, "Oh, promptly notified means they're going to get back to me in two days." They may be sitting across the table saying, "Promptly notify means we're going to get back to them in a month." You need to make sure these things are defined.
You also want to make sure that anything you're signing doesn't rely or reference other agreements that can be changed at any time. You may sign a contract that says, "This contract can only be changed by mutual agreement, with 30 day notice, and signature of all parties involved." But that contract might rely a terms of service or a service level agreement that can be changed at any time.
You want to make sure that anything that is part of the contract is there attached to the contract and everything is held to the same standard. We talked about the integrator. You want to see the contract that the integrator has with the provider. You want to know that their obligations are the same as their obligations to you.
You want to know who's involved. I've seen lots of contracts that say, "We're going to provide our affiliates with information." It's like, who are these people? Who are the affiliates? They're not signatories to this contract. What access are they going to have to my information? I want to know these things.
You have to know these things, because you cannot outsource your responsibility. There is no way you're going to be able to answer a Congressional inquiry when your information has been compromised and you're going to say, "Oh, Google's affiliates did it." What the heck? No one's going to take that seriously. You can't outsource responsibility.
You have to present all aspects of the technologies to the decision makers, the pros and the cons. Because if anything is withheld, it's going to be impossible to defend the decision. You can still make a business decision to move to the cloud, acknowledging that there are risks. There's risks with everything. There's risk waking up in the morning and crossing the street.
Of course, we are not paralyzed by risk. But we need to make sure that we are aware of it and we're doing what we can to mitigate it or accept it. If you have a document that says the CIO has analyzed all the risk and he or she is accepting them, fantastic because hey, we're acknowledging them. But you cannot bury your head in the sand.
I appreciate you eating the broccoli today. I hope that you guys have learned some stuff. I have one question that's come in through the Q and A. [beep] If anybody else has any questions, feel free to ask or post them in the Q and A.
My information is at the end, as is Trey's. Feel free to contact us if you need anything. My email address is a little bit different because I'm on detail right now [beep] to the Special Inspector General's office, but I should be back around the end of January, but I'm happy to answer any questions.
The first question that I have was, "Do you have a sample cloud government service level agreement or a good cloud contract?" I get asked that everywhere. I have lots of examples of bad ones and I'm happy to help you negotiate a good one, but the thing is that an example of a good contract is going to depend on the parties involved.
The requirements for the Department of Interior are going to be a little different than the Department of State and are going to be a little different than the International Trade Commission. I can give you an example. There's actually some great clauses that the Inspector General community has negotiated for IG access. You could probably start there for some cloud computing clauses.
But a full contract, I don't have anything that's perfect from soup to nuts, because each one's going to be different. But I'm happy to give you some input into what a good contract would contain. The second question is, how come the DOI Google contract is not ATO'd? [beeps]
I don't know why the Google contract is not ATO'd. I don't really know the history of how you guys got to sign on the dotted line with Google Cloud. I know that they are a super effective sales and marketing force.
I've spoken, I've given this presentation in a lot of different environments, and I actually gave this presentation once, and there was a guy, he looks like Rick Moranis, from Google. He's like the head of Google Federal. He literally looks like Rick Moranis. I'm not joking.
He came up to me afterwards and was like, [whining] "Why are you picking on us? It's not fair." I was like, "OK, tell me what I said that [beeps] wasn't accurate." He just started whining some more.
Google's super good at selling stuff. I don't know how or what happened at Interior that it was signed, and I don't know if any of these concerns were taken into consideration. Again, because I'm not a DOI lawyer, I don't know. Maybe you can talk to somebody in the solicitor's office. That's all I've got on that question.
What do we do with [laughs] current cloud contracts? The question is, what do you do if you're stuck with something? Here's what I would do. There's a couple things you can do.
Number one, you can take the "Questions to Ask" document that you guys all have and you can do a program review. Do an internal program review and see if what they're providing you is answering these questions. See if what they're providing you is giving you a comfort level in your risk mitigation.
That is one thing you can do. If you want to go rogue, you can take the "Questions to Ask" to your Inspector General's office, and you can ask them to do a post-procurement review. They can...sometimes, I'll tell you what. The IG is not an organization to be afraid of. Sometimes if the IG writes a report it gives management cover to take action that they wanted to take all along.
Maybe they don't have senior management buy in, but maybe they're like, "Look. We know that we need to do this, and now the IG has even told us we need to do it. It will get done." Sometimes that's a good approach to take, but if you have a current contract, and you realize that there's holes in it you can always modify it, but I will warn you that it's going to cost you more money.
As soon as you sign on the dotted line, and you want to mod a contract it always comes with additional cost. I would suggest be prepared to pay more. How do we approach executive management, and both bureaus and departments with this information?
Very carefully. I think what you need do is you need to find senior management, number one that gets it, or number two that is willing [beep] to sit there and say "I don't get it, but I don't want to be testifying in front Congress." Scare tactics work well a lot. You can find a lot of these examples of cloud computing gone awry, and you can do the house of horrors.
That can help scare them sometimes. I think the way you approach it is you say, "Look. We know that you want to do this. We know that the Administration was pushing this, but now that Vivek's gone, I don't know what's going on. We understand that there may be some cost savings, but we want to make sure that we're doing this correctly."
Always being the first kid in the pool is not necessarily the best way to approach it. What I would do, honestly, is I would go back to questions one and two that I presented earlier. I would say, "Is there a business plan?" Review the business plan again and say, "Is it sufficient?"
Maybe they can say, well, there was a business plan, it wasn't well thought out. Maybe we're deploying this now and maybe it needs some tweaks. Maybe you won't be able to put the toothpaste back in the bottle, but maybe with some tweaks and some restrictions, it may be something that's manageable and you may be able to live with.
But because I don't know the details behind interior, you'd really want to know who the players are and what they're getting into. Our contract secret, how do we get copies? OK, this is actually a great question. Who, Lydia, I promise you that, yeah, Lydia's not placed.
At the beginning of this year, GSA had an open house with the cloud providers and customers. They had all the cloud providers in a room, it was the integrators, it wasn't the big guys. There were probably about 10 or 12 of them. [beeps] I went to each and every table and I introduced myself.
I gave them my card. I said, "Hey guys, can you send me a copy of your standard contract clauses?" At first, they were like, "Oh, each contract's unique and we negotiate with the customer," and blah blah blah. I was like, "Look, I don't...I know that. Give me your standard clauses. What's your incident response? What's your arbitrary, the standard boilerplate stuff that you put in every contract."
I asked every single one of them and gave every single one of them my card and asked them all to send me just the standard boilerplate clauses. None of them got back to me. Not even a, "Oh, Sabrina. We talked with legal and legal says we can't release it." Nothing. Silence. Which again, to me, did not give me great confidence in these vendors, it was actually frightening.
I think that they believe that these contracts are secret. I am from Florida, the Sunshine State. I am a big fan of transparency. Sunlight's the best antiseptic or whatever they say. Anything that is too secret to see, I want nothing to do with. If the contracts are being treated that way, then we've got a problem because you need to know exactly what you're buying and how you're buying it.
If they're not willing to tell you, walk away. Run away, as quickly as you can. All right, so we've got another. There aren't any official FedRamp authorized systems yet, but there are some that were provided provisional ATO's via GSA.
Well, according to the FedRamp website, none of them have been given provisional ATO's. Cloud services operating [indecipherable 57:49] requirements. Maybe. At one of the conferences I went to, I had...The FedRamp guy was up on the panel and the Google guy was harassing him from the audience insisting that Google had been approved at the high level and the FedRamp guy was saying, "We don't even think you've been approved at the moderate level."
I remember, if some of you guys were reading in the trade news, there was a big kerfuffle between Google and Microsoft. Google was going around saying that they had been FISMA approved. Microsoft was saying that they were lying. I happen to believe Microsoft's version of the story, especially since the FedRamp guy was insisting that Google hadn't even been preliminarily ATO'd.
I don't know. Maybe GSA in their wonderful contract negotiating with them told people that they had a provisional ATO, but the website right now says nobody's been given provisional ATO's. That's what we're going with. Why is getting a copy of the integrator's contract with the provider so important?
This is important, again, like I said because you need to know what the integrator's responsibilities are with the prime, with the big guy, the guy that's providing the cloud service because the integrator is not providing anything but a middleman, holding your hand, how to transfer your information to a cloud environment. That's all they're doing. They don't run the server farms.
They don't run the security, that's for sure. Again, like I said, you want to make sure that you know what the prime big guys requirements are with the integrator, that you know how that impacts you. The biggest thing I can think of is incident response. If you want to know what's going on with your information and you insist that you know within 24 or 48 hours, that's great. But if the integrators...