NGINX - Part 1 - Reverse Proxy Configuration

1_ Reverse Proxy with NginX

The topology for this test:

<Client>  ====>  [Nginx - Reverse Proxy] ====> [Backend Server]

1) Installation (on both server Proxy and Backend)

#sudo apt-get update
#sudo apt-get install nginx

2) Configuration 

Backend server (Using Nginx also)

After install nginx, edit nginx.conf and put some line

#sudo vi /etc/nginx/nginx.conf

Content of file:

user www-data;
worker_processes 4;
pid /run/nginx.pid;
worker_rlimit_nofile 30000;

events {
        worker_connections 10240;
        # multi_accept on;
}

http {

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;


        ## Start: Timeouts ##
        client_body_timeout 10;
        client_header_timeout 10;
        keepalive_timeout 5 5;
        send_timeout 10;
        ## End: Timeouts ##


        #server_names_hash_bucket_size 128; # this seems to be required for some vhosts
        index    index.html index.htm index.php;


        include /etc/nginx/sites-enabled/*;
}

Save the file and move to edit virtual host:
Change directory to /etc/nginx/sitess-available, create a file with the name as what you want and fill some thing (as sample below):

server {
        listen 8888;
        server_name  backend.nhut.com;
        root /web_data;

location / {
        include block-country.conf;
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
}

}

This configuration will create a website working under named "backend.nhut.com" with root dir site at /web_data and port listen on port 8888. Some next line, just inside "location" will make your site working as "directory listing".

This purpose of configuration is that we will do a Proxy Stress test using cosbench (or Jmeter, ab... as which as you want). Under /web_test, we will create a directory structure for tressing test purpose.

After create web virtual host, create a link to enable site

#sudo ln -s /etc/nginx/sites-available/nhut_backend /etc/nginx/sites-enabled/nhut_backend

and start nginx

#sudo /etc/init.d/nginx start

You should not got error ^^ for this step (checking error log : #sudo tail -f /var/log/nginx/error.log).


Proxy server (Nginx)

Install Proxy server on a separate server. Before we get into Nginx configuration, first thing, we should check our server limit configuration for advance test. Run command

#ulimit -Sn
#ulimit -Hn

If the number is too small, we should change it as below procedure:
1) Open the file /etc/security/limits.conf with your right permission, then paste the following line towards end:

*         hard    nofile      500000
*         soft    nofile      500000
root      hard    nofile      500000
root      soft    nofile      500000

You can change the number "500000" as you want. And REMEMBER: Once you save file, you may need to logout and login again.

2) Open /etc/pam.d/common-session, Add following line:

session required pam_limits.so

//Just forget it if it already had

3) Open /etc/sysctl.conf. Add the following;

fs.file-max = 2097152

And then run:

#sysctl -p

After this step, you can verify the new limit.
Now, move to configure Nginx working as Proxy server:

Edit /etc/nginx/nginx.conf and paste below configuration:

user www-data;
worker_processes 4;
pid /run/nginx.pid;
worker_rlimit_nofile 30000;

events {
        worker_connections 10240;
        # multi_accept on;
}

http {

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;


        ## Start: Timeouts ##
        client_body_timeout 10;
        client_header_timeout 10;
        keepalive_timeout 5 5;
        send_timeout 10;
        ## End: Timeouts ##


        #server_names_hash_bucket_size 128; # this seems to be required for some vhosts
        #index    index.html index.htm index.php;


        # Cach configure
        #limit_conn_zone   $binary_remote_addr  zone=nhutcache:10m; 
        proxy_cache_path /proxy_test/ levels=1:2 keys_zone=nhutcache:100m max_size=300000m inactive=30d; #if inactive value is reached, Cache-status in ngxin cache log will be MISS

        proxy_temp_path /proxy_test/tmp;


        log_format cache 'BEGIN-- ***$time_local '
                     '***Cache-status: $upstream_cache_status '
                     '***Cache-Control: $upstream_http_cache_control '
                     '***Remote-addr: $remote_addr '
                     '***Expires: $upstream_http_expires '
                     '***Action: "$request" ($status) ($body_bytes_sent Bytes) '
                     '***HTTP-referer: "$http_referer" '
                     '***Client-Browser: "$http_user_agent" --END';


        access_log  /var/log/nginx/cache.log cache;

        #Test if command

        #if ($status = ^(403|404)$ ) {
        #access_log off;
        #}
        include /etc/nginx/sites-enabled/*;
}

In nginx.conf we just create a memory caching allocation named "nhutcache" with cache directory on local disk at /proxy_test. So, we should create that directory for store the cache files (and directory for Proxy_temp_path as well).

Next, create virtual site for Nginx connection:
Change directory to virtual site /etc/nginx/sites-available, create a file and paste following :

upstream proxytest {
        server 10.76.0.46:8888;  # this is IP of backend server that we just create on step above

                }

server { 
        listen 8888;
        server_name  proxy.nhut.com;



        #for static file
        #location ~* ^.+.(jpg|jpeg|gif|png|ico|css|txt|js|htm|html|avi|mpeg)$ {
                location ~* .(jpg|jpeg|gif|png|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|tar|wav|bmp|rtf|swf|ico|flv|txt|xml|docx|xlsx|js|css)$ {
                include proxy.conf;
                #include block-country.conf;
                proxy_pass  http://proxytest;
                proxy_cache nhutcache;
                proxy_cache_valid 200 302 301 304 7d;   #this config for expiration of statis files with response 200 302 ...
                #expires 5m;
                #return 500;
                }
      

#for any queries to site that begin with "file"
location ~ ^/file {
                include proxy.conf;
                #include block-country.conf;
                proxy_pass  http://proxytest;
                proxy_cache nhutcache;
                proxy_cache_valid 200 302 301 304 7d; #if this value (7d) is reached, Cache-status in nginx cache log will be EXPIRED
                #return 500;
                }


        #For all remain queries
                location / {
                include proxy.conf;
                #include block-country.conf;
                proxy_pass      http://proxytest;
                proxy_cache nhutcache;
                proxy_cache_valid 200 302 301 60m;
                proxy_cache_valid 404 403 1m;
                #return 500;
                        }
}

Note, this file we will include 2 file from outside is proxy.conf and block-country.conf, so that we should create both file and put in same dir with nginx.conf, the content of both file is below:

Proxy.conf

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_ignore_headers Expires Cache-Control;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffer_size       4k;
proxy_buffers           32 4k;
proxy_busy_buffers_size    64k;
proxy_temp_file_write_size 64k;

Block-country.conf (this file is optional, i put this because in this test, i use public IP for some case)

allow <IP of range/mask>;
allow ......./24;
allow 127.0.0.1;
deny all;

Create a link to enable your site:

#sudo ln -s /etc/nginx/sites-available/nhut_proxy /etc/nginx/sites-enabled/nhut_proxy

Now, start Nginx

#sudo /etc/init.d/nginx start

If you want to mornitor Nginx cached, just:

#sudo tail -f /var/log/nginx/cache.log

If you want to show HIT/MISS/EXPIRE status, just run:

#sudo awk '{print $5}' /var/log/nginx/cache.log  | sort | uniq -c | sort -r