1. Home
  2. Training Library
  3. Amazon Web Services
  4. Courses
  5. Creating a Highly Available Campaign Website - Scenario

Defining Technical Requirements

play-arrow
Start course
Overview
DifficultyBeginner
Duration2h 7m
Students457

Description

defining technical requirements for loungebeer.com

Feb 2018 - Amazon DynamoDB now supports encryption at rest. This was a requirement we had to work around in this scenario. This new feature would make selecting DynamoDB a far easier choice for the AWS solution.
https://aws.amazon.com/about-aws/whats-new/2018/02/amazon-dynamodb-now-supports-server-side-encryption-at-rest/

Transcript

- All right, so we'll go through and define some technical requirements, and I think if we work to just the four key areas so we can think about our client, like, what are you thinking, Ben? What will be our sort of front-end for this? Is this gonna be an SDK? Is it gonna be a client?
- We'll go responsive.
- Right,
- [Ben] just HTML, CSS, JavaScript.
- [Developer] Before we get too deep in the technical aspect is there anything else we need from John?
- Yeah, anything we haven't thought of?
- Guys. Anything?
- What about disaster recovery? Say if there's some huge disaster, is there anything that we, what would happen if this whole US east coast went down under an earthquake or something like that?
- [Developer] You mean besides we're all fired? I dunno.
- [Developer] I think if all of the eastern seaboard goes down we have bigger concerns.
- Yeah.
- Yeah.
- [Developer] The point is well-made.
- Okay, cool, Alright, so.
- Well we can just build that into the higher variability depending on the design we choose we can look at high resiliency, and multi-region, et cetera,
- Okay. so we can build that.
- [Andrew] Yeah, John, I'm pretty confident we can do this for you and I think we can bring it in under those two time and cost constraints.
- Great. So we'll try and evaluate the best platform to use to deliver on those requirements. We might need to do a proof of concept, so ideally we can come up with a design in the next three or four hours and then go back go back to your customer and say, "Look, for two or three K we can build a little proof of concept to prove one or three or two of these solution platforms can deliver the type of environment we need for this."
- [John] Okay.
- Would that be something you could sell them or would you want us to just to come out with a hard recommendation and go for it?
- Well let's give them a few options.
- Okay, all right, great. Alright, so we'll probably go for an evaluation matrix. So what we'll try and do here is define our technical requirements so we'll do those in those four buckets. What the front end needs to be, what the web application or the application layer needs to be, what our database layer needs to be and maybe what our reporting or export needs to be. Alright, so anything else on the client? Shall we work top down or bottom up? What do you prefer?
- Top down?
- Top down, okay. All right then, what about security here? What do you recommend, do we need to do anything on, we're using https for capturing--
- [Developer] Yeah, yeah, yeah. End to end encryption if security is that big of an issue.
- [Developer] Can we let John go now?
- [Developer] Yeah, John, you don't need your eyes to glaze over.
- [John] You were starting to lose me there.
- Yeah we could see you getting pretty bored with us already, so.
- Talk to you in a bit.
- Yeah, okay.
- Cheers, thanks John.
- Give us a couple hours, we'll come up with a solution for you. Alright, so, Stuart, thinking data in transit needs to be protected.
- [Stuart] Uh huh.
- So in terms of like his security requirements, he was talking about needing it to be secure. So what type of, what sort of protection do we need to put at the client, http level traffic?
- [Stuart] With web traffic we should have a CDM.
- Yep, okay cool.
- [Stuart] Cause I think although it's only once only in the stadium, that's not gonna happen. Realistically, that's, that's
- No, you're right. That's not gonna happen so we need to cater for a wider audience.
- Yep.
- I think he's got some contradictory requirements. He's got a million plus requests.
- [Andrew] Yeah.
- But he's saying just in the stadium ...
- [Andrew] Yeah.
- Which tells me the customer doesn't really know what they want.
- [Andrew] They're gonna want us to run this for more than the month. It's gonna be like a 12 month campaign for sure.
- Yeah.
- So we might as well think about the top end.
- Yeah. It's also probably gonna get slammed for a much shorter time frame and we'll need to really deal with a lot of burst traffic with this one, I'd say.
- [Developer] With security being an issue we can incorporate a web application firewall.
- Okay cool, yeah. So we put that at the application level perhaps. So we'll put an application firewall. Now, in terms of how we deal with this burst activity I'm thinking is it multi-region, multi-availability zone? What are your thoughts?
- [Developer] Yeah, because one region can cover the stadium as multi AZ but if some traffic comes from outside the stadium...
- Yeah, okay so the multi AZ. Does it need to be multi-region?
- [Ben] I think we should.
- [Stuart] I think so
- Yeah. To follow along with what Marko just said.
- Right.
- [Stu] He did kinda bang on about my availability quite a lot so ...
- Yeah. Alright so ...
- [Stuart] But we can give him the two options.
- Yeah.
- The cost for both multi-region ...
- [Ben] Yes.
- Or single region and multi AZ.
- [Andrew] Yep, cool okay. Now, so between our client and our application this is gonna be a HTTP post so we're submitting it in transit securely.
- [Stuart] Yeah.
- We've possible got some posts coming through CDN and viral web application firewall so we've got a little bit of protection. But do we need to do anything else to this data that's being posted to it to make it more secure?
- [Ben] Yeah, so once the object gets posted you want to make sure you white list those properties so that should someone go and modify that post request and and a novel to it, it's not basically taking up all your space in the database causing a denial of service.
- Okay, perhaps we should just sketch out these layers first and then go back
- Yeah. to the sort of detail. I'm jumping all around here aren't I. So, in terms of our app layer, I'm sorta started thinking serverless. What do you think?
- [Ben] Same.
- Same?
- [Ben] Yeah.
- Yeah.
- Yeah? Okay, alright, so we don't need to spin up any servers for this. This is going to be something that we can run through a managed service. And if we go back to our kind of requirements and our sort of environmental factors like where we're doing multi-regional, multi-availability zone. Let's try and use as many managed services as possible. Because that means that the availability will be taken care of for us which will make things a bit easier, right?
- Yeah.
- But with serverless solution you need it to think about a pre-warming tasker that can spin up all the service.
- Right, good point. Because with serverless functions we do have that container model which, you know, sometimes does need to be pre-warmed doesn't it? And because we've got very, very high spikey traffic here where we need to be able to scale very quickly.
- [Developer] Yeah. To deal with that type of, you know, possible through put. So we need to have some sort of pre-warming or function that keeps the ...
- Yeah. the serverless layer really to scale. Okay, now what are your thoughts on how we manage requests from the app layer to, to whatever our database is to.
- [Stuart] We're gonna need some kind of curing system.
- Right.
- Just so we don't lose any data.
- Yeah.
- It's gonna be a huge amount of traffic.
- [Andrew] Yeah.
- So we wanna kinda decouple those two components.
- Good thinking. Cause he said that it was, we must capture the stuff.
- [Stuart] Yeah.
- Can't be lost. So, if we use a cue where we can reply with, you know, if it's five or ten minutes later, the customer's gonna get an email confirming their ...
- [Stuart] Yeah.
- Interest, et cetera. Alright, cool. And if the cue is another scaled managed service which again gives us a little bit more availability. Alright, so what about our database layer?
- The knee jerk reaction with something like this, something this scalable, is no sequel.
- [Andrew] No sequel.
- But it's the encryption part that's kinda throwing me.
- [Andrew] Yep.
- So we're talking encryption at rest, yeah?
- [Andrew] Yeah.
- What do we have for options ...
- From, from an
- Cross platform. AWS perspective DynamoDB doesn't offer encryption at rest.
- [Andrew] Yeah, at present we'd have to go up to Aurora or Post Grace to get encryption at rest.
- Okay, so we can ...
- [Ben] We can consider like client side.
- Yep, we can definitely ...
- [Ben] We can do it at the, not a, not client as in the front end, but client as in the API layer?
- Yep.
- Yep.
- [Stuart] But we could also do it with something like a sequel database like you're saying.
- Sorry, just staying on the, yeah ...
- How about from a Azure and Google front?
- For no sequel?
- Yeah.
- Google has encryption for data store by default, and a ...
- Okay.
- What about Azure?
- Azure has the same, same feature as AWS. So the no sequel database is not encrypted.
- [Ben] Not encrypted.
- While a sequel has been encryption.
- [Ben] Okay.
- Hey, that's a good point.
- We're kinda heading towards, unless we're going to do it at the API, we're heading towards a sequel option.
- [Stuart] And that's similar with AWS as well. With the audio service, it's a simple tick box to encrypt data at rest.
- [Ben] Fair enough.
- [Stuart] And that applies to your backups, your snapshots.
- [Ben] Okay.
- Right, right. It does perhaps stretch this corner doesn't it? If we're going for a audio solution that may be a little bit more.
- [Ben] Yeah. So, ideally we want a no SQL database but we're probably gonna have to use some sort of relation database, right?
- [Ben] Yep.
- Okay. We can put these into our options. Maybe that's part of our proof of concept is to see if we can use the ah ... Cause the AWS DynamoBD service has a STK that we can use to encrypt data on the client side before we save it. But it's gonna get pretty messy. Hmmm, okay, just on those API's, so what does our API need to do? If we got a, we got a serverless app layer. We've got a stateless database. We're gonna need some, you know, pretty clear verse scalable API's in the middle. Any specifics we need for that?
- The API is gonna be fairly simple. All we're doing is taking a post request. This is a, this is a read only.
- [Andrew] Yep.
- It's read heavy, but it's a read only app. So ah ...
- [Andrew] Write only.
- My apologies, write only.
- Yeah. Yeah, so it, it's pretty light weight. If we wanna be able to write something cross platform would the three major cloud vendors we're looking at Node JS.
- [Andrew] Uh huh.
- If we're, if we narrow it down to AWS and Azure we could also include a C-sharp and some other stuff, so.
- Cool, alright.
- [Stuart] Maybe keep it open to Node at this point then so we can run it across all three.
- [Ben] It gives us more options.
- Yep, alright. So let's go Node or C-sharp.
- [Ben] Nice. Our app layer. Alright, okay, so how about the monitoring? Do we need to get clever with that? If we're gonna keep this available at all times.
- We do need to have some operational monitoring there to...
- Yep.
- If there any issues of failures that it's, that it's failing over correctly.
- [Andrew] Okay, alright. So we'll make that part of our assessment.
- And some notification alerting as well.
- Yeah, alerts to us, yeah. He didn't say anything about how he wants the data exported so ...
- [Ben] That gives us ah ...
- Gives us a little room to work. Yeah it does.
- [Ben] Maybe a little wiggle room there.
- Alright then, let's just quickly try and sketch out some services that we could look to evaluate from each of the three platforms to deliver this architecture. So, Stew a few ...
- [Stuart] I got a column there for AWS.
- Yeah, there we are. So for ... CDN will go will go route 53.
- [Stuart] For DNS and CloudFront.
- CloudFront.
- [John] For the CDN.
- Marko, what would we use for a Azure?
- [Marko] You can use traffic manager.
- Yeah.
- [Marko] As a for DNS. And a, and a CDN for the CN service.
- [Andrew] Okay, Ben, anything from Google?
- Yeah, we have CDN and DNS.
- [Andrew] Alright, cool. Alright, no point of difference there then. And for our app layer, app layer we use lambda serverless functions from AWS. And functions from Azure.
- [Marko] Yes.
- [Ben] Functions from Google as well.
- Right. Cool. And code base we're going Node JS or C-sharp. Can we run Node JS on, in a Google function?
- [Ben] Yep.
- Alright, okay imagine if we could come up with an application layer.
- [Ben] Yeah, that's what Stew was saying. Maybe we pick Node JS
- Yeah. And make it fairly generic use and some interfaces to kinda run cross platform.
- Cool. And in terms of our security, like we got WAF for AWS.
- [Ben] How's that play in? Lambda and WAF?
- Yeah, well ...
- [Ben] What's that look like?
- A little bit of work to do there, I think.
- [Ben] But it is possible?
- It's possible, yeah.
- Okay. And just because he's given us that quite specific constraint about not being hacked or have any kind of ejection issue or anything of that like, so ...
- [Ben] Yeah.
- The way the application firewall, with working in conjunction with CloudFront anyway, can you know, give us a little bit more protection about that and reduce the amount of D-DOS text we might have to, ya know ...
- [Ben] Now that I think about it, we're using a cue, so ...
- The cue's gonna add another cushion isn't it?
- [Ben] Well, so, what I don't even know is, we need an application firewall necessarily.
- Alright.
- What we're doing is the post comes into the API.
- Yeah.
- And it dumps it onto the cue. There are very few points of injection there.
- [Andrew] Yeah, okay.
- No, sequel injection because the, another function it's, you know, kinda hidden. That's gonna be processed in the cue.
- [Andrew] Nice.
- So that handles the sequel.
- [Andrew] So that probably could become, a should.
- But if there is any kind of injection at any point before that, then it can be stripped at the, at the CDN level.
- [Andrew] And remember we've got a, we're gonna have to deal with any attack situation really quickly.
- Our biggest attack factor is denial of service.
- Yeah, so if someone hits this really hard during that peak time.
- [Ben] Yeah.
- We've gotta have a service to at least absorb or mitigate it. You know, if we were using EC2 here Stew, I'd put an ELB in there. Cause an elastic load balance will give us just a little bit of extra layer. But using lambda, you know, what other controls do we need? We should think this through.
- I do think it is worth having the WAF in there though.
- [Andrew] Yeah, yeah.
- I see what you're saying, but I think we should ...
- [Andrew] Let's leave it ...
- Cause those rules will be pushed out to every edge location.
- Okay.
- And on the CloudFront so ...
- Sure.
- It hits it straight in there, the first point.
- Let's see how it plays out in cost as well. Cause if it adds an extra layer ...
- Especially on the D-Dos as well.
- Yeah, if it gives of denial of service protection then we should probably put it in.
- Yeah.
- Yeah.
- Alright, have we got a similar service from Azure and Google? What would that web application file work?
- Yes, we have application gateway.
- [Andrew] Okay, right.
- If we're talking application gateway, how does that play in with functions through, Marko?
- Well, it works at https level, so ...
- We have to set up our own sub-net through application gateway into that?
- Yep.
- And then do some additional routing.
- Yeah.
- A little more work. We, yeah.
- Yep.
- [Andrew] And we may not get a lot of value from that. So I think we'll leave it as a possible should.
- Yeah.
- Okay, so the cue will go SQS for Amazon.
- [Stuart] What about Google?
- Ah yes.
- [Stuart] On the application firewall.
- What do we got for a firewall in Google?
- [Ben] I'm not, so, functions are still in preview as it it and anything for WAF ... We'll look into that one. I'm not sure there's any good scenario that would integrate.
- [Stuart] Just the, just jumping back to the DNS element did John say that this has to stay within the US? I'm just thinking and geo-restriction.
- He didn't say, but ...
- [Andrew] He, he probably ...
- For security requirements what are we talking about there?
- [Andrew] Yeah, we may need to like geo block. We certainly want to do some latency based routing.
- I think that's maybe something we should bring up here.
- We should talk about lot about that.
- He will hit us with that.
- [Stuart] There's threats at that level.
- Yep, good thinking.
- [Ben] Good idea.
- So our database, we're going, did we decide if we're going to do encryption or not?
- [Ben] That's good question. I think we'll have to ... Let's spec it out for, for both. We'll do a proof of concept for both and than see what happens.
- Okay, right. But, it sounds like we're all talking no se--, sorry, sequel options because out of the gate we get encryption.
- Okay, mmm hmm. So we go DynamoDB, Aurora, or Post Grace. It's probably on this side.
- [Stuart] How does that work with multi-availabilities and...
- Yeah.
- [Stuart] And multi-region for those databases?
- Yeah, we'll have to do a better design around that if we're going to multi-Z.
- [Stuart] Cause if we use Audi S then it's very simple.
- Yeah, it would make it easy for us, wouldn't it?
- But I'm not too sure how it works with those.
- Cause the cost difference between Post Grace and Aurora and DynamoBD, its not going to be massive. But it does feel like a bit of an over cook, doesn't it? For just these records.
- [Stuart] Yeah.
- However, if we do get the benefit of encryption plus multi-AZ support, it may be worth, just, you know, pushing that into the cost corner because we get a lot more from it.
- [Stuart] Yep.
- Yeah, and there's not a real benefit from having no SQ apart from the speed. The speed is something that we know it can handle that type of activity very well. We know even the cue fits.
- [Ben] But the cue kind of mitigates that for us.
- Yeah, yeah, yeah.
- [Ben] Enough that.
- Yeah.
- [Stuart] It may not any real value in this app.
- Okay, good thinking. Alright so, what about Azure for database.
- For database you have a document DB.
- [Andrew] Yeah.
- No sequel offering. While you have sequel database for the sequel offering.
- Okay, and what have we got for Google Ben?
- Cloud Sequel and Cloud Data Store.
- Okay, now did I hear you say that the no SQL version of Google has encryption?
- [Ben] Yes. Yeah.
- That's a big plus. Yeah, okay. Alright, yeah what we'll have to do a price comparison I think. We stand them up.
- [Ben] And then some load testing to see how our sequel options
- Yep. Are gonna kinda stand up. Make sure the cues really do it's job.
- That's gonna be, gonna be good. Alright, so for monitoring we go Cloud Trail, Cloud Watch, and anything else Stew?
- [Ben] Stew, you were talking about Cloud Trail the other day, we were having a conversation.
- [Stuart] Yeah,
- There's a conflict?
- Yeah, so, hmm, tell me, do we need Cloud Trail? You were talking about it for auditability.
- For auditing and kinda governance control, it allows you to see exactly ...
- Right.
- What, who called an API.
- Do you think that's gonna be an issue for this?
- Source IP, internet service or user.
- [Andrew] I absolutely do. Cause I think this gonna have to be ...
- This is a contest where we might have some auditing.
- [Andrew] Yep.
- Some sort of ...
- [Andrew] Yep, they'll have to draw it out. We'll have to give a full record at some point of any activity that went on there so.
- [Stuart] So, Cloud Trail will capture it all.
- [Ben] Nice, that's great.
- Would you use Conflict too Stew? Would you put Conflict in there?
- I would probably stick Conflict in there as well. Say if you got a record of whose creating what resources, if any resources change.
- [Andrew] Yeah.
- If any resources get terminated. And that also links in with Cloud Trail as well. So for any kind of configuration item it will have a link to the API call so you can see exactly who or what service.
- That's great.
- [Andrew] That covers every possible question we could get around a compromise or a data loss of some sort.
- [Stuart] If there's any compliance as well from the customer a link to a deity.
- Yep.
- [Stuart] Then that will capture pretty much.
- [Ben] That's awesome.
- Nice one. Alright, so what do we put for Azure on here? What do we got?
- [Marko] You have Azure monitor that can do that work.
- Okay, cool, and Google?
- [Ben] Stack driver.
- Stack Driver.
- [Ben] Yep.
- Alright, well so far the only difference has been around encryption really, isn't it? Like the encryption baked in to no SQL stack with Google. Which is good. And alerting would go, like, we got Cloud Watch alarms, Stew?
- [Stuart] Yep, Cloud Watch alarms, SNS.
- And if we do this white list as suggested we can build some lambda functions to actually change or update any IP addresses or change any rules on the fly. Again, I think that speed of how quickly we need to adjust things, to make sure that everything's always available is going to be critical here. Anything else for alerts for Azure or Google?
- [Marko] Also, monitor has some alerting features.
- Okay.
- [Ben] Stack Driver, again.
- Yeah, alright, cool.
- [Stuart] SNS for IDS.
- Aww yeah right, good thinking.
- [Stuart] And you could also pipe the logs from Cloud Trail into Cloud Watch as well. I know some monitoring there.
- That's awesome.
- So if there's any API's that should be called we can be notified right away.
- Nice.
- Now, John probably doesn't know it yet but the client's bound to want some sort of reporting, right? So, how do they know if this has even been a success and importantly, how do we prove that we've delivered it as they want it?
- [Ben] Sure.
- So, is there any type of reporting layer we could put in here?
- [Ben] How are we quantifying? What's that look like?
- Yeah. I'm talking about reporting just raw numbers of requests.
- Yeah.
- [Ben] Or is it something more?
- Yeah.
- [Ben] Without knowing that it's really tough to say but...
- Should we leave it off the list then? Like we make it, like a perhaps, a should or a could. Because you're right. Yeah, we're just adding extra functionality, which ...
- [Ben] I put it as a should or a could until we have that more info then we can kind of flesh that out a bit.
- Yeah, alright. Technology wise, I was just thinking that could be a great use of Quick Site, which is the new AWS reporting tool. So you can basically just do a kind of, like, but you can do it with BI.
- [Marko] We have power BI.
- [Andrew] Have power BI, yeah, yes. Alright, so if we need that we do have some great tools available. I think Google's got a pretty good one too as I call. Anyone remember?
- [Ben] I don't recall off the top of my head what they have.
- [Andrew] Look into it. Okay, so just quickly to close.
- [Stuart] Sorry just before you do, he mentioned about the real time monitoring.
- Ah, so it was some real time monitoring.
- [Stuart] You said we could drop that off as a feature if the cost becomes too much. I just quickly, a high level, from an AWS perspective, if we could use Kinesis for real time, real time.
- [Andrew] Real time.
- Analytics. Okay, nice.
- Due to the cost constraints is that something we really want to build in house versus software as a service?
- Yeah. Yeah, good question.
- Well we can implement Kinesis and then have third party software just pulling out the data.
- That's great then.
- As a round kinda front end dashboard.
- Nice.
- Have we missed anything?
- Probably a lot, yeah.
- However, this gives us a strawman we can stand up and test out. So at least if we give top level design we can do a price calculation and make sure that we can come in within that. I think all of those three solutions would come in within four weeks.
- [Ben] Yeah.
- Yeah?
- [Ben] Yeah.
- Yeah.
- [Stuart] Well if we gonna go in in with an AWS solution
- Yeah, yeah. If you can knock up a Google design ...
- Sure.
- And how it would look in Google and for yourself Marko, Azure. And then if we meet again and we run through.
- Yeah.
- How it looks.
- [Andrew] Test them out.
- Sure. Good one. Okay, 

 

About the Author

Students51612
Courses77
Learning paths28

Andrew is an AWS certified professional who is passionate about helping others learn how to use and gain benefit from AWS technologies. Andrew has worked for AWS and for AWS technology partners Ooyala and Adobe.  His favorite Amazon leadership principle is "Customer Obsession" as everything AWS starts with the customer. Passions around work are cycling and surfing, and having a laugh about the lessons learnt trying to launch two daughters and a few start ups.