Lan Party Resources

From kipiki
Jump to: navigation, search

Caching services

I've utilized a few resources to make a caching services for the following services (http only, https is passed through, explained later):

Content delivery networks I'm hosting:

  • Steam
  • Arenanet
  • Battle.net
  • Frontier
  • GOG
  • Origin
  • Riot
  • Rockstar
  • Sony
  • UPlay
  • Wargaming
  • Microsoft Windows Updates

Project resources I used

https://github.com/steamcache/generic - Caching server, from what I gather, basically a ubuntu shell with a nginx/squid reverse proxy that caches
https://github.com/steamcache/sniproxy - Proxies https traffic without decoding it, basically bounces it to the destination without modification
https://github.com/uklans/cache-domains - Maintained list of domains that services use for their CDN
dnsmasq - for local dns handling

Set up DNS

DNS is a very important component to this, you need to redirect traffic that is going to certain addresses to your content caching server.
Template:WarningSetting up the DNS service and failing to set up the caching services and SNI Proxy will make services fail to be able to update. You must

I use dnsmasq as my dns service, if you use something else (most people at home have a hardware router, not a linux box like I do) you may have to choose a different method but the resources provided will provide you will all of the material you need. This is one of the many advantages of running a 'nix box as your router.

On your dnsmasq server, add the following line to the end of your dnsmasq.conf to allow it to read a second file for configuration:

## this may be different depending on your os, this is for debian linux
vi /etc/dnsmasq.conf

## at the end of the file, add:
conf-file=/etc/dnsmasq_cachehosts.conf

I wrote a handy script (you should run as root, users should not be changing config for services like dnsmasq that have root privs) to grab the material fresh from the uk-lan github and then populate the dnsmasq file using the caching server of 10.0.0.7 (change ip if yours is different).
Description of what this script does:

  • uses wget to grab each of the desired caching lists and put into a file (in this case /root/cachedns)
  • writes a newline behind each entry, some of the lists do not have a return character after the last entry and it makes two entries on one line
  • Sed then:
    • Removes any comments, we don't want those as config items
    • Removes any blank lines (many of which we just created), we don't want any blank items, that would be exceedingly bad (think it would forward *all* traffic to our destination)
    • removes the *. and . in front of many of the entries, dnsmasq takes entries like example.com to mean anything.example.com so any subdomain under example.com would be grabbed, so far this behavior has not presented any issues.
  • zeros the config file we are modifying (why we are doing it as a secondary file)
  • uses a for loop to create the new config file
  • restarts dnsmasq to re-read the config file (may not be necessary, not sure)

Create (then later) change permissions on script to be rwx for owner only.

vi cachedns.sh

## after editing
chmod 700 cachedns.sh

Script Contents:

#!/bin/bash
wget https://raw.githubusercontent.com/uklans/cache-domains/master/steam.txt -O - > /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/arenanet.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/blizzard.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/frontier.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/gog.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/origin.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/riot.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/rockstar.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/sony.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/uplay.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/wargaming.net.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
wget https://raw.githubusercontent.com/uklans/cache-domains/master/windowsupdates.txt -O - >> /root/cachedns
echo "" >> /root/cachedns
sed -i 's/#.*$//g' /root/cachedns 
sed -i '/^$/d' /root/cachedns
sed -i 's/^\*//' /root/cachedns
sed -i 's/^\.//' /root/cachedns
>/etc/dnsmasq_cachehosts.conf
for i in `cat /root/cachedns`; do echo "address=/$i/10.0.0.7" >> /etc/dnsmasq_cachehosts.conf; done
/etc/init.d/dnsmasq restart

And now just run the script and you will have a nice new config file and the content pointing at the content server:

./cachedns.sh

Resulting config file would look like this:

# cat /etc/dnsmasq_cachehosts.conf 
address=/content.steampowered.com/10.0.0.7
address=/content1.steampowered.com/10.0.0.7
address=/content2.steampowered.com/10.0.0.7
address=/content3.steampowered.com/10.0.0.7
address=/content4.steampowered.com/10.0.0.7
address=/content5.steampowered.com/10.0.0.7
address=/content6.steampowered.com/10.0.0.7
address=/content7.steampowered.com/10.0.0.7
address=/content8.steampowered.com/10.0.0.7
address=/cs.steampowered.com/10.0.0.7
address=/steamcontent.com/10.0.0.7
address=/client-download.steampowered.com/10.0.0.7
address=/hsar.steampowered.com.edgesuite.net/10.0.0.7
address=/akamai.steamstatic.com/10.0.0.7
address=/content-origin.steampowered.com/10.0.0.7
address=/clientconfig.akamai.steamtransparent.com/10.0.0.7
address=/steampipe.akamaized.net/10.0.0.7
address=/edgecast.steamstatic.com/10.0.0.7
address=/steam.apac.qtlglb.com.mwcloudcdn.com/10.0.0.7
address=/assetcdn.101.arenanetworks.com/10.0.0.7
address=/assetcdn.102.arenanetworks.com/10.0.0.7
address=/assetcdn.103.arenanetworks.com/10.0.0.7
address=/live.patcher.bladeandsoul.com/10.0.0.7
address=/dist.blizzard.com/10.0.0.7
address=/dist.blizzard.com.edgesuite.net/10.0.0.7
address=/llnw.blizzard.com/10.0.0.7
address=/edgecast.blizzard.com/10.0.0.7
address=/blizzard.vo.llnwd.net/10.0.0.7
address=/blzddist1-a.akamaihd.net/10.0.0.7
address=/blzddist2-a.akamaihd.net/10.0.0.7
address=/blzddist3-a.akamaihd.net/10.0.0.7
address=/blzddist4-a.akamaihd.net/10.0.0.7
address=/level3.blizzard.com/10.0.0.7
address=/nydus.battle.net/10.0.0.7
address=/edge.blizzard.top.comcast.net/10.0.0.7
address=/cdn.zaonce.net/10.0.0.7
address=/cdn.gog.com/10.0.0.7
address=/wpc.11df.deltacdn.net/10.0.0.7
address=/11df-eu-lb.wpc.edgecastcdn.net/10.0.0.7
address=/11df-eu-lb.apr-11df.edgecastdns.net/10.0.0.7
address=/origin-a.akamaihd.net/10.0.0.7
address=/akamai.cdn.ea.com/10.0.0.7
address=/lvlt.cdn.ea.com/10.0.0.7
address=/river.data.ea.com/10.0.0.7
address=/l3cdn.riotgames.com/10.0.0.7
address=/worldwide.l3cdn.riotgames.com/10.0.0.7
address=/patches.rockstargames.com/10.0.0.7
address=/pls.patch.station.sony.com/10.0.0.7
address=/gs2.ww.prod.dl.playstation.net/10.0.0.7
address=/gs2.sonycoment.loris-e.llnwd.net/10.0.0.7
address=/cdn.ubi.com/10.0.0.7
address=/dl.wargaming.net/10.0.0.7
address=/dl2.wargaming.net/10.0.0.7
address=/wg.gcdn.co/10.0.0.7
address=/wargaming.net.edgesuite.ne/10.0.0.7
address=/wgusst-na.wargaming.net/10.0.0.7
address=/wgusst-eu.wargaming.net/10.0.0.7
address=/update-v4r4h10x.worldofwarships.com/10.0.0.7
address=/dl-wows-ak.wargaming.net/10.0.0.7
address=/download.windowsupdate.com/10.0.0.7
address=/officecdn.microsoft.com/10.0.0.7

If something goes arwy

If for some reason, your content server below doesn't work, or something is screwy, you can remove the config line at the end of /etc/dnsmasq.conf:

conf-file=/etc/dnsmasq_cachehosts.conf

or zero the file you created with the script:

> /etc/dnsmasq_cachehosts.conf

Then restart dnsmasq

/etc/init.d/dnsmasq restart

Set up content server

The server I'm using for caching uses a nfs mount /export/steamcache for its material storage. That is set up using NFSv3 and zfs on the fileserver side with a 2TB limit and compression lz4 turned on, setting up a separate fileserver is not covered in this document. We will be using docker for the caching server and the proxy setup, setting up docker is not covered in this document. Just letting you know because it is interesting design variety.
Docker requires you to remove the container, the reason we have the -v options are to save the logs and data off to some non-container directory so when we update we don't lose all of the material when we do updates later on.

Setting up a secondary ip for the content server

the content/proxy needs to use 80/443 ports and if you use the server for webservices or whatnot you need those ports, this may not be necessary for your setup

## example is for debian linux, range 10.0.0.0/24, your network settings will vary
## I have my machine set up with 'sane' old ethernet names, and use interfaces file, setting that up is not covered here
vi /etc/network/interfaces

## assuming you are using eth2 as your primary network device, you can add another ip by just adding :1 and adding more config like so:
auto eth2:1
iface eth2:1 inet static
netmask 255.255.255.0
address 10.0.0.7

Setting up Caching Docker container

This setup assumes you are using, adjust values as necessary:

  • /export/steamcache/allcache as your base directory (and that it exists, docker will create the other directories)
  • destination directory has at least 2TB of space (default is 500000m)
  • the system has 4G memory to allocate to the container (default is 500m)
  • the ip that its binding to is 10.0.0.7
docker pull steamcache/generic:latest
docker create --name cache_all --restart unless-stopped -p 10.0.0.7:80:80 -v /export/steamcache/allcache/logs:/data/logs -v /export/steamcache/allcache/cache:/data/cache -e CACHE_MEM_SIZE=4000m -e CACHE_DISK_SIZE=2t steamcache/generic:latest
docker start cache_all

Setting up SNI Proxy

This setup assumes you are binding to 10.0.0.7, adjust as needed

docker pull steamcache/sniproxy:latest
docker create --name sniproxy --restart unless-stopped -p 10.0.0.7:443:443 steamcache/sniproxy:latest
docker start sniproxy

Checking the service for activity

You can look at the service through docker or directly from your mounted logs directory.
Here is how you would tail the live logs from docker. HIT means that it got the item from the cache (locally), MISS means it had to reach out to the remote service to get the material (then it cached it):

 # docker exec -it cache_all tail -f /data/logs/access.log
10.0.0.180 - - [29/Dec/2017:19:34:58 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "MISS" "blzddist1-a.akamaihd.net" "bytes=254300248-254566487"
10.0.0.180 - - [29/Dec/2017:19:34:58 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "blzddist1-a.akamaihd.net" "bytes=253142216-253408455"
10.0.0.180 - - [29/Dec/2017:19:34:58 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=254566488-254832727"
10.0.0.180 - - [29/Dec/2017:19:34:58 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=255098968-255365207"
10.0.0.180 - - [29/Dec/2017:19:34:58 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=255365208-255631447"
10.0.0.180 - - [29/Dec/2017:19:34:58 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "blzddist1-a.akamaihd.net" "bytes=254832728-255098967"
10.0.0.180 - - [29/Dec/2017:19:34:59 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "blzddist1-a.akamaihd.net" "bytes=255631448-255897687"
10.0.0.180 - - [29/Dec/2017:19:34:59 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "blzddist1-a.akamaihd.net" "bytes=256163928-256430167"
10.0.0.180 - - [29/Dec/2017:19:34:59 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 17435 "-" "-" "HIT" "blzddist1-a.akamaihd.net" "bytes=256430168-256447602"
10.0.0.180 - - [29/Dec/2017:19:34:59 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=255897688-256163927"
10.0.0.180 - - [29/Dec/2017:19:34:59 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "MISS" "level3.blizzard.com" "bytes=258063921-258330160"
10.0.0.180 - - [29/Dec/2017:19:35:00 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=258746819-259013058"
10.0.0.180 - - [29/Dec/2017:19:35:00 +0000] "GET /tpr/d3/data/ed/cc/edcc675e778c3e579eae9620b272cb86 HTTP/1.1" 206 266240 "-" "-" "HIT" "blzddist1-a.akamaihd.net" "bytes=259013059-259279298"
...

Maintaining the services

On occasion you will want to update the containers and dns entries, you may want to even schedule these in cron.
If scheduling in cron, I would recommend not doing more then once a day for dns, and probably once a week for the docker containers.

Updating Caching Docker container

This assumes same material as setting up, see notes on setup above.

docker stop cache_all; docker rm cache_all; docker pull steamcache/generic:latest
docker create --name cache_all --restart unless-stopped -p 10.0.0.7:80:80 -v /export/steamcache/allcache/logs:/data/logs -v /export/steamcache/allcache/cache:/data/cache -e CACHE_MEM_SIZE=4000m -e CACHE_DISK_SIZE=2000g steamcache/generic:latest
docker start cache_all

Updating SNI Proxy Docker container

This assumes same material as setting up, see notes on setup above.

docker stop sniproxy; docker rm sniproxy; docker pull steamcache/sniproxy:latest
docker create --name sniproxy --restart unless-stopped -p 10.0.0.7:443:443 steamcache/sniproxy:latest
docker start sniproxy

Updating the dnsmasq entries

Just run the script you made on the router/dns server:

## probably as root
./cachedns.sh