Build Servers That Runs Forever

With cloud services like AWS EC2, you can quickly set up a Linux server and run software on it, like LAMP/LEMP, Node.js, etc.  However, to run a server for months or years, there are a few things you can do to minimize required maintenance.

Usually Linux servers can run for long time without having to reboot or cleanup, unless the software you run on it consumes too much disk or memory over time.

If you run a webserver like Apache or Nginx, they by default enables request/error logs, in most cases you don't have to worry about it, but theses logs do grow over time.  If you do not cleaning them up regularly, they could take over much of the disk space and cause issue with operating system or other software.

Rotate logs regularly

Use the logrotate utility provided by Linux to keep only the latest logs and automatically delete older logs.  You only need to write a simple config file and the system service will automatically clean up logs based on your config.  E.g. only keep the last 7 days of compressed logs and remove all older logs.

Separate data partition

Even with logrotate, we may still have significant disk usage by logs or other data files.  One common problem with cloud machines is the root partitions are small and only have a few GBs of free space.   Once the free space is used, the operating system may have problem running basic services.  So if you anticipate more than a few GBs of disk usage, it is always a good idea to mount an extra partition dedicated for data usage (and leave enough free space in the root partition for operating system).

You may also consult your software manual to move log files to separate data partition, E.g. for Apache web server, you can update the httpd.conf entries to tell it create the log files in different paths.

Memory Usage

Besides disk space, memory usage can also grow over time and cause system crashes.  Most people would reserve enough intial memory for their server and software to start running.  However, you may need more memory over time, especially if you server deal with accumulative data (e.g., load more data into memory when user data grows after a few months).

It is important to examine your data/memory usage for some time to get a good sense of how much it would grow over time, and reserve enough memory to deal with the worse case or have a scaling strategy.

There are more things to consider when building a server that supposed to run 24x7 for a long time, but start with things above will give you a good basis.

Create Static Website Using AWS (Part 2)

In the first part of this tutorial, we showed you how to host a static website on your own domain name using AWS S3.  In this second part, we will show you how to add CloudFront (AWS CDN service) on top of it.  Static websites on S3 usually loads fast already, however having a CDN can provide many additional benefits, including better performance and scalability, lower cost, fine grain control of compression and client caching, and in this case adds HTTPS support.

Prerequisites:

• Everything we did in part 1, so you have a static site at http://www.acme.com
• Access to DNS entries of your domain so we can modify them. Your Domain Registrar (like GoDaddy, NameCheap, etc.) usually provides easy UI to do this.

Steps:

1. Go to CloudFront console and create a new Distribution, use the S3 domain created in part 1 as the "Origin domain", e.g., "www.acme.com.s3.us-east-1.amazonaws.com").
2. Leave most of the settings as default, scroll down to "Price class" setting, and choose "Use only North America and Europe".  CloudFront has edge servers all over the world, and they are priced differently.  This change makes sure you only use the lowest priced locations even if a user from other continent visits your site.
3. Add your own domain name in the "Alternate domain name (CNAME)" list, e.g. www.acme.com
4. Request a certificate for your domain in the "Custom SSL certificate".  This will redirect you to the AWS Certificate Manager (ACM) page, and please follow instruction to submit and validate the request.  Make sure you add both acme.com and *.acme.com in your request.
5. Click "Create distribution" to finish.
6. Now you should see the new distribution in the Distributions list.  Click its name to view details and copy the "Distribution domain name".  It should look like "d1234abcd.cloudfront.net"
7. Go to your DNS provider to modify the CNAME entry we created in part 1, so that "www.acme.com" points to the Distribution domain name you copied above.
8. Wait a few minutes for DNS entry to propagate and you should be able to access your static website through CloudFront CDN.

With the above steps, when a user visits http://www.acme.com, it first route the user to CloudFront server; CloudFront will request the file from S3 and return it to the user, and at the same time store it in a cache. So that next time anyone requests the same file, CloudFront will read it from cache and return to users, rather than read from S3 again.

Besides, with the certificate, users can visit your site through HTTPS like  https://www.acme.com

Create Static Website Using AWS

Modern websites are more dynamic than ever, yet you may still be surprised what a static website (simple directory of html/css/javascript/image files) can do.  Not only does it require minimal set up, it's also super fast with a simple CDN on top.

In this tutorial, we will show you how to host a static website using AWS services like S3 and CloudFront.

Prerequisites:

• AWS account.  If you don't already have it, is it super easy to sign up at https://aws.amazon.com/account/sign-up It also comes with free tier quota so you can play with the services without having to pay a penny.
• Static website in a local directory.  For the purpose of this tutorial, we assume your website is in a local directory called mysite/  and there are the following files in it:
• index.html
• style.css
• main.js
• favicon.ico
• mypic.jpg
• A desktop app to upload your local website directory to S3, you can use CyberDuck or other similar tools.  You can also use AWS CLI command-line tool if you feel comfortable with it.
• (Optional) If you want your website to use your own domain name, you need to register a domain name with your own Domain Registrar (like GoDaddy, NameCheap, etc.). For easier explanation, let's assume your domain is acme.com

Steps:

1. Go to S3 console and create a new S3 bucket, you should use the domain name as the bucket name.  E.g., in this example, bucket name should be "www.acme.com"
2. Make sure you disable "Block public access" setting for the new bucket
3. Make sure you disable "Block public access" setting at your S3 account level as well.  You can still enable this setting for specific buckets if you like.
4. Enable "Static website hosting" for your new bucket, choose "Host a static website" as Hosting Type, and type "index.html" as Index Document, and Save
5. Once you done the above, you should find the "bucket website endpoint" at the bottom of your new bucket's properties tab.  Note down this URL for later use.
6. Go to the permissions tab of your new bucket and edit the "Bucket Policy", fill in something like this (replace the bucket name with your own bucket name)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::www.acme.com/*"
        }
    ]
}

7. Now your new bucket is ready, upload your static website directory to the new bucket using CyberDuck.  You should configure CyberDuck connection with your own AccessKeyId and SecretAccessKey (they act like username and password to connect to S3).  You can find them under your AWS account's Security Credentials or AWS IAM
8. After upload, you should see be able to see the pages from browser using the "bucket website endpoint" URL noted before, e.g., http://www.acme.com.s3-website-us-east-1.amazonaws.com/
9. (Optional) if you want to access the website through your own domain name like http://www.acme.com, you would need to create a CNAME entry with your DNS provider that points "www.acme.com" to the bucket website endpoint "www.acme.com.s3-website-us-east-1.amazonaws.com".  Please consult your DNS provider on how to do this, most likely your domain registrar already provides a simple web UI for your to edit the DNS entries.
10. Now you can access your static website with URL like http://www.acme.com

Congratulations!  You just created a static website, now share it with your friends :).

In the next part we will discuss how to add a CDN on top of this static website.

Let's build smarter code!

To make better software!