Jump to content

Native Linux server (with management scripts)


Alloc

Recommended Posts

Sorry guys, if I'm Saying something redundant. I searched on many of these 52 pages to find my answer, and didn't find the right solution.

 

My Server is crashing and shutting down the game application. On ubuntu community, someone said it would be an lack of telnet response and it match with some forum users problems. Someone created a cron script to check if the server is running, to schedule a restart and start the server again in cases of crashes?

 

I repeat, I'm only ask it, cause I really didn't find it on this community. If someone has the answer, please: can you draw it to me?

 

I assume my ignorance. Even if you have the answer, can you tell me how to make the cron task?

 

Ahn... I forgot!

 

I'm running the server on a VPS, with 7,2 Ghz and 3 Gb of Ram, and I'm planning to migrate to a VPS SSD 2, with 1 vCore, 2.4 GHz and

4 GB RAM...

Edited by Fernando_Torres (see edit history)
Link to comment
Share on other sites

Your server is most likely crashing due to a lack of RAM. I really cannot stress enough that for a dedicated server, you must have 8GB to expect to run it, and 12-16GB if you want it to run well. (32-bit Linux is limited to 4GB of RAM usage, but you must have all 4GB of RAM available and not shared with the system. Even then, I would not recommend that for more than 4 people on a Navezgane map)

 

Are you renting a server, or do you have your own hardware?

Edited by SylenThunder (see edit history)
Link to comment
Share on other sites

Your server is most likely crashing due to a lack of RAM. I really cannot stress enough that for a dedicated server, you must have 8GB to expect to run it, and 12-16GB if you want it to run well. (32-bit Linux is limited to 4GB of RAM usage, but you must have all 4GB of RAM available and not shared with the system. Even then, I would not recommend that for more than 4 people on a Navezgane map)

 

Are you renting a server, or do you have your own hardware?

 

 

I'm Renting a server... today I searched on HTOP, and realized I has 2 instances Running. Then I deleted the other instance and I'm waiting to see what happens. It lowered the RAM usage in 1 gb.

 

I'm intending to look for a new and cheaper VPS now... this is the result of my htop. Thank you SylenThunder

print.jpg.afcdb030e0c0b8d02a2fcdb172f9c05d.jpg

print.jpg.4661e44d2bffbab77284a4c99ee13b7d.jpg

Edited by Fernando_Torres
I messed the image (see edit history)
Link to comment
Share on other sites

Just to update. Despite the crashes (lol) the server is running with four to five players so far so good.

 

I found an VPS with 8 gb, on OVH. How many players you recommend to me? I'll keep you informed by the performance, and lets see what happens.

I would keep it under 10. Technically MP only supports 8, but I can do 20 with 16GB RAM on a dedicated Linux blade. Once we get past 15, and a lot are exploring, there is a bit of latency though. (Which truly, could be a result of the very high active zed settings I have, and not the number of players.)

Link to comment
Share on other sites

Well, yeah, you are running the 32 bit server ;)

Ok dangit. I commented it out to run in 64-bit. So I guess that's not working.


Ok, I went and double-checked, and as far as I can see, it should be loading 64-bit.

 

Here's what my startserver.sh looks like.

 

#!/bin/sh

cd "`dirname "$0"`"

 

 

PARAMS=$@

 

 

CONFIGFILE=

while test $# -gt 0

do

if [ `echo $1 | cut -c 1-12` = "-configfile=" ]; then

CONFIGFILE=`echo $1 | cut -c 13-`

fi

shift

done

 

 

if [ "$CONFIGFILE" = "" ]; then

echo "No config file specified. Call this script like this:"

echo " ./startserver.sh -configfile=serverconfig.xml"

exit

else

if [ -f "$CONFIGFILE" ]; then

echo Using config file: $CONFIGFILE

else

echo "Specified config file $CONFIGFILE does not exist."

exit

fi

fi

 

 

export LD_LIBRARY_PATH=.

#export MALLOC_CHECK_=0

 

 

if [ "$(uname -m)" == "x86_64" ]; then

./7DaysToDieServer.x86_64 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS

else

./7DaysToDieServer.x86 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS

fi

 

 

So if that's not working, how do I fix it? :bull_head:

Edited by SylenThunder (see edit history)
Link to comment
Share on other sites

Ok dangit. I commented it out to run in 64-bit. So I guess that's not working.


Ok, I went and double-checked, and as far as I can see, it should be loading 64-bit.

 

Here's what my startserver.sh looks like.

 

#!/bin/sh

cd "`dirname "$0"`"

 

 

PARAMS=$@

 

 

CONFIGFILE=

while test $# -gt 0

do

if [ `echo $1 | cut -c 1-12` = "-configfile=" ]; then

CONFIGFILE=`echo $1 | cut -c 13-`

fi

shift

done

 

 

if [ "$CONFIGFILE" = "" ]; then

echo "No config file specified. Call this script like this:"

echo " ./startserver.sh -configfile=serverconfig.xml"

exit

else

if [ -f "$CONFIGFILE" ]; then

echo Using config file: $CONFIGFILE

else

echo "Specified config file $CONFIGFILE does not exist."

exit

fi

fi

 

 

export LD_LIBRARY_PATH=.

#export MALLOC_CHECK_=0

 

 

if [ "$(uname -m)" == "x86_64" ]; then

./7DaysToDieServer.x86_64 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS

else

./7DaysToDieServer.x86 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS

fi

 

 

So if that's not working, how do I fix it? :bull_head:

What does "uname -m" return?

Link to comment
Share on other sites

"uname -m" checks the machine Hardware [x86_64" or "x86"] .

so when you have a x64 machine it runs the x64 client else the x86 client.

Yeah, the question was targeted to Sylen to see if it indeed does return the expected value on his machine ;)

Link to comment
Share on other sites

What does "uname -m" return?

Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-77-generic x86_64)                                                                                                                                                                                  
* Documentation:  https://help.ubuntu.com/                                                                               

 System information as of Sun Feb 21 16:19:32 EST 2016                                                                   

 System load:  0.01               Processes:           118                                                               
 Usage of /:   73.6% of 48.72GB   Users logged in:     0                                                                 
 Memory usage: 7%                 IP address for eth0: 192.168.0.143                                                     
 Swap usage:   0%                                                                                                        

 Graph this data and manage this system at:                                                                              
   https://landscape.canonical.com/                                                                                      

38 packages can be updated.                                                                                               
0 updates are security updates.                                                                                           

Last login: Sun Feb 21 16:19:10 2016 from 192.168.0.5                                                                     
root@7Days2Die:~# uname -m                                                                                                
x86_64                                                                                                                    
root@7Days2Die:~#                                                                                                         

Which is why I'm confused now. It used to run 64-bit just fine.

Link to comment
Share on other sites

...

Which is why I'm confused now. It used to run 64-bit just fine.

Yeah, weird ...

You could either try debugging the script (e.g. adding some echo's to see where it's going and what it's doing) or just remove the if-stuff for platform detection altogether. But it will most likely not show up in the server browser then ;)

Link to comment
Share on other sites

Hi,

 

my Server randomly crashes when connecting to the web interface. I know this some time now but didn't manage yet to copy the log file into the forum.

 

2016-02-22T17:00:50 265747.915 INF Steam certificate error: RemoteCertificateChainErrors
2016-02-22T17:00:51 265748.839 INF Steam OpenID login from 217.225.77.226:62797 with ID 76561197972223708, permission level 0

 

This are the last entry's before the server stops working. In htop its still running but with ...

 

7dtd.sh status OZCOOPII

 

it keeps telling its offline...

 

 

Instance: OZCOOPII

Status: NOT running

Game info:
   Server name:    [DE] Omega-Zirkel [COOP][PUBLIC TS3][sIRMOD]
   Password:
   Max players:    8
   World:          Random Gen

 

Seems all network connections are gone away. Any idea ?

 

UPDATE:

 

After htop F9 and (re)start i connected to the web gui again and looked at the output log. The certificate error is also in there but not stopping the server.

 

2016-02-22T17:42:50 2239.520 INF Steam certificate error: RemoteCertificateChainErrors
2016-02-22T17:42:51 2240.490 INF Steam OpenID login from 217.225.77.226:64067 with ID 76561197972223708, permission level 0
2016-02-22T17:43:09 2258.718 INF Web:IconHandler: Icons loaded - 5961 ms


 

Greetings,

Devidian

Edited by Devidian
more info (see edit history)
Link to comment
Share on other sites

OK. I don't get it.

 

I modified startserver.sh to

#if [ "$(uname -m)" == "x86_64" ]; then	./7DaysToDieServer.x86_64 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
#else
#	./7DaysToDieServer.x86 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
fi

And yet the log is still showing 32-bit.

Link to comment
Share on other sites

OK. I don't get it.

 

I modified startserver.sh to

#if [ "$(uname -m)" == "x86_64" ]; then    ./7DaysToDieServer.x86_64 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
#else
#    ./7DaysToDieServer.x86 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
fi

And yet the log is still showing 32-bit.

 

Going to have to start a fresh box up and test on it I guess.

 

Edit: New box, fresh install, and still I get...

2016-02-22T21:19:41 0.087 INF Version: Alpha 13.8 (b7) Compatibility Version: Alpha 13.8, Build: Linux 32 Bit

2016-02-22T21:19:41 0.087 INF System information:

2016-02-22T21:19:41 0.129 INF OS: Linux 4.2 Ubuntu 15.10 64bit

>.<

Edited by SylenThunder (see edit history)
Link to comment
Share on other sites

my Server randomly crashes when connecting to the web interface. I know this some time now but didn't manage yet to copy the log file into the forum.

...

The output regarding the certificate is "normal" behaviour. Also doubt the webserver part itself is directly responsible for the crash, more likely some sideeffect in combination with the core code or just something simple like OOM. Did you have a look at the *top* of the log file? Depending on what's causing problems the Linux client sometimes overwrites the content there with error messages.

 

 

 

I modified startserver.sh to

#if [ "$(uname -m)" == "x86_64" ]; then    ./7DaysToDieServer.x86_64 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
#else
#    ./7DaysToDieServer.x86 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
fi

I guess the line breaks are broken in this copy? Otherwise the whole exec of the server would be commented out ;)

 

 

Edit: New box, fresh install, and still I get...

2016-02-22T21:19:41 0.087 INF Version: Alpha 13.8 (b7) Compatibility Version: Alpha 13.8, Build: Linux 32 Bit

2016-02-22T21:19:41 0.087 INF System information:

2016-02-22T21:19:41 0.129 INF OS: Linux 4.2 Ubuntu 15.10 64bit

>.<

That's definitely weird. Just tested over here and starting the 64 bit build will definitely output "Build: Linux 64 Bit"

Link to comment
Share on other sites

Yeah, I'm certain something is goofy. Yes it broke the line breaks in the copy.

 

I can't figure out why it's doing this on two different systems though. Sadly, by newly-created test platform crashed the kernel today and I cannot get it back up. Either the hardware is fried, or I dunno. don't feel like messing with it. I think I'll just copy the 64-bit file and rename it to the 32-bit file for now. LOL.

 

Edit: Nope. that didn't work. I start it now, and it says "Failed!". Even tried flipping the comments around in startserver.sh. I've got a real case of weird going on here.

 

Ok, I'm stupid. permissions issue. It's up again, but I can still only get 32-bit for some stupid reason.

Edited by SylenThunder (see edit history)
Link to comment
Share on other sites

The output regarding the certificate is "normal" behaviour. Also doubt the webserver part itself is directly responsible for the crash, more likely some sideeffect in combination with the core code or just something simple like OOM. Did you have a look at the *top* of the log file? Depending on what's causing problems the Linux client sometimes overwrites the content there with error messages.

 

You are right, the error is on top of the log file, funny :o

Here it is:

 

[s_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.Setting breakpad minidump AppID = 251570
Stacktrace:


 at (wrapper managed-to-native) UnityEngine.Object.Destroy (UnityEngine.Object,single) <0x00004>
 at (wrapper managed-to-native) UnityEngine.Object.Destroy (UnityEngine.Object,single) <0x00004>
 at UnityEngine.Object.Destroy (UnityEngine.Object) <0x00023>
 at AllocsFixes.NetConnections.Servers.Web.Handlers.ItemIconHandler.LoadIcons () <0x008bb>
 at AllocsFixes.NetConnections.Servers.Web.Handlers.ItemIconHandler.HandleRequest (System.Net.HttpListenerRequest,System.Net.HttpListenerResponse,AllocsFixes.NetConnections.Servers.Web.WebConnection,int) <0x00022>
 at AllocsFixes.NetConnections.Servers.Web.Web.HandleRequest (System.IAsyncResult) <0x003c0>
 at System.Net.ListenerAsyncResult.InvokeCallback (object) <0x0004f>
 at (wrapper runtime-invoke) object.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <0x00046>


Native stacktrace:


       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x8960f) [0xf6e6660f]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x21a63) [0xf6dfea63]
       [0xf77b8420]
       /home/sdtd/engine/libstdc++.so.6(_ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_+0x98) [0xf76fe9f8]
       /home/sdtd/engine/7DaysToDieServer.x86() [0x81fe106]
       /home/sdtd/engine/7DaysToDieServer.x86() [0x81fd8cf]
       /home/sdtd/engine/7DaysToDieServer.x86() [0x8201ef8]
       /home/sdtd/engine/7DaysToDieServer.x86() [0x85444b3]
       [0xf5cc33f0]
       [0xf5cc33ac]
       [0xead63d6c]
       [0xead632cb]
       [0xebc4d5a9]
       [0xebc4d1d0]
       [0xf6b440bf]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x25a1b) [0xf6e02a1b]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(mono_runtime_invoke+0x66) [0xf6f03aa5]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(mono_runtime_invoke_array+0x53b) [0xf6f084f6]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x12b822) [0xf6f08822]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x153ca4) [0xf6f30ca4]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x15413e) [0xf6f3113e]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x153b0d) [0xf6f30b0d]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x17d0fb) [0xf6f5a0fb]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86/libmono.so(+0x19e868) [0xf6f7b868]
       /lib32/libpthread.so.0(+0x6f59) [0xf7795f59]
       /lib32/libc.so.6(clone+0x5e) [0xf7577c4e]


Debug info from gdb:




=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================


IP address: 127.0.0.1
IP address: 85.214.212.32
eac_server.so [x86] :: OnLoad()
STEAMPS3 - AsyncTCPSocket created
STEAMPS3 - AsncTCPSocket destroyed
STEAMPS3 - AsyncTCPSocket created
STEAMPS3 - AsncTCPSocket destroyed


 

Another thing, i thought about building my own WebInterface (with node.js/socket.io etc.) and how nice it would be to have a native game "web-sockets" like api that runs on localhost:xxx. So web-service and game service would be divided in 2 tiers. I could look into it myself and make my own mod but 1. its long time ago i was into C# (currently only in web based languages) and 2. it would be cool to have something like this from "official" side ;)

What do you think about this ?

Link to comment
Share on other sites

OK. I don't get it.

 

I modified startserver.sh to

#if [ "$(uname -m)" == "x86_64" ]; then    ./7DaysToDieServer.x86_64 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
#else
#    ./7DaysToDieServer.x86 -logfile 7DaysToDieServer_Data/output_log__`date +%Y-%m-%d__%H-%M-%S`.txt -quit -batchmode -nographics -dedicated $PARAMS
fi

And yet the log is still showing 32-bit.

 

Going to have to start a fresh box up and test on it I guess.

 

Edit: New box, fresh install, and still I get...

2016-02-22T21:19:41 0.087 INF Version: Alpha 13.8 (b7) Compatibility Version: Alpha 13.8, Build: Linux 32 Bit

2016-02-22T21:19:41 0.087 INF System information:

2016-02-22T21:19:41 0.129 INF OS: Linux 4.2 Ubuntu 15.10 64bit

>.<

 

If you are starting the server with Alloc's scripts then I don't think it uses the startserver.sh file.

This script is hard coded to use the 32-bit build /usr/local/lib/7dtd/commands/start.sh

 

The server works okay in 64-bit but does not show in serverlist.

Log shows:

INF [steamworks.NET] GameServer.LogOn timed out

Link to comment
Share on other sites

If you are starting the server with Alloc's scripts then I don't think it uses the startserver.sh file.

This script is hard coded to use the 32-bit build /usr/local/lib/7dtd/commands/start.sh

 

The server works okay in 64-bit but does not show in serverlist.

Log shows:

INF [steamworks.NET] GameServer.LogOn timed out

Editing startserver.sh always worked before without issues to put it in 64-bit. (Since a12, I've run more stable on the 64-bit than 32.)

 

Tested with editing line 60 in /usr/local/lib/7dtd/commands/start.sh. It does load the 64-bit version.

 

So the real question is, what changed? Because I never had to edit this file to run the 64-bit with Alloc's scripts before.

Link to comment
Share on other sites

I would keep it under 10. Technically MP only supports 8, but I can do 20 with 16GB RAM on a dedicated Linux blade. Once we get past 15, and a lot are exploring, there is a bit of latency though. (Which truly, could be a result of the very high active zed settings I have, and not the number of players.)

 

I changed the server, and hosting on OVH, with 2,4, 2v cores and 8 gb RAM, It runs fine with less than 10 players. I crashed only one time, but I saw a high use of CPU. Thanks for the help. I'll return if the crashes continue to happens.

Link to comment
Share on other sites

Stacktrace:
 at (wrapper managed-to-native) UnityEngine.Object.Destroy (UnityEngine.Object,single) <0x00004>
 at (wrapper managed-to-native) UnityEngine.Object.Destroy (UnityEngine.Object,single) <0x00004>
 at UnityEngine.Object.Destroy (UnityEngine.Object) <0x00023>
 at AllocsFixes.NetConnections.Servers.Web.Handlers.ItemIconHandler.LoadIcons () <0x008bb>

Are you running any mods on that server? Especially additional items / item icons?

No idea why it would crash in the destroy method unless it's already having memory issues at that point.

 

 

Another thing, i thought about building my own WebInterface (with node.js/socket.io etc.) and how nice it would be to have a native game "web-sockets" like api that runs on localhost:xxx. So web-service and game service would be divided in 2 tiers. I could look into it myself and make my own mod but 1. its long time ago i was into C# (currently only in web based languages) and 2. it would be cool to have something like this from "official" side ;)

What do you think about this ?

You mean like the API it already has and the web frontend uses to get the required data? ;)

 

 

 

So the real question is, what changed? Because I never had to edit this file to run the 64-bit with Alloc's scripts before.[/color]

Nothing ever changed on the Linux scripts regarding the started executable. It's directly invoked the .x86 executable since the very first release for the native Linux 7dtd build on 2014-06-02.

You either edited the scripts start-script or you didn't use my scripts to start the server or you never actually ran the 64 bit build ;)

Link to comment
Share on other sites

Are you running any mods on that server? Especially additional items / item icons?

No idea why it would crash in the destroy method unless it's already having memory issues at that point.

No, just vanilla. I have a xui mod running some weeks now but the crashes like this were happening before too. Not often but they happen.

 

You mean like the API it already has and the web frontend uses to get the required data? ;)

ؤhm,.. wait... no. Currently it uses xhr requests (polling) not web-sockets. Web sockets is a more real time connection, the client gets a permananet connection and the server pushes any changed data. More like publish/subscribe...

 

More like (pseudocode)

 

ws = connect(localhost:port).

...

ws.on('playerList',dosomething);

..

ws.emit('subscribeToPlayerList');

...

 

:D

 

edit: by the way, can't you configurate the web-server to push an empty transparent map tile on missing files ? the console 404 warnings ... (i dont like that :D)

I know how to configurate nginx to do that but as you use your own ...

Edited by Devidian (see edit history)
Link to comment
Share on other sites

Ok now i'm home from office i may explain a little bit more what i would like to have.

 

The goal behind that is splitting web and game server, so if the game server stops working the webserver would still run and say "instance not running" if the webserver crashes on the other side, the game server is still running.

 

Another benefit would be having only 1 webserver that may handle multiple sources (gameserver) on the same machine.

There may be 2 ways i'm currently thinking how i would realize this. The first i call "data-dump" where the game-server just dumps all data to a special directory that could look like:

 

data\players.json // all exposed player data

data\entitys.json // all current alive entitys (zombies/animals)

data\map\* // exported map like currently (with the tiles as images, maybe as copy from the original exported map)

data\*.json // everything else to expose, like claims, logs, server (instance) configuration, player permission levels, etc

 

Then on the (external) web service there could be configured the base path to instances directory so it could automaticly scan for all status.json files in the instance data directory's or the pathes could manually configured. In node.js for example i could use file change watcher that always reload the data on any file changes. I wont go more into details here, because this isn't my preferred way.

 

=====

 

A Better solution would be what i call "web-socket-service" that may also handle user permissions. So the server runs something like socket.io for node.js (only built-in) where a web-service (either another backend process, or a frontend lib like socket.io) can connect to and register with the steam id. Then for example i could join rooms like "player.location" that pushes any player location changes on change to the room (where multiple user could be registred to) if i have the right permission level. Also i could with the right permissions emit commands to the server or join rooms like "chat" where i can follow the game chat live.

This way only the map tiles would need to be created and maybe copied (to not access the original files that frequently are write accessed by the game)

 

In both scenarios there wont be any web-server overhead per instance, just a way to push the data either to files or to web-sockets.

 

I hope my english was good enought to explain my thoughts - otherwise i prefer german :D

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...