Server failed to start after backup failed (FS full on self-hosted server)

[DarkEnd]

Bonjour,
 

Version Server : Ubuntu 22.04
Version Game Server : Alpha 21
Test To Validate files via Steam => YES

 

Log_console_last_launch : https://pastebin.com/xXveQjR9

J'ai mis en place une sauvegarde chaque lundi de mon serveur. sauf que le système de fichiers était plein... La sauvegarde a évidemment échoué 

Normalement, le serveur s'arrête, sauvegarde et démarre... Mais le serveur n'a jamais redémarré.😞 

 

Je démarre le serveur en mode débogage pour avoir des erreurs, et voici ce que j'ai : 
 

2023-10-24T14:31:42 36.991 INF Loading and creating biomes took 967ms
2023-10-24T14:31:43 37.895 INF Loading and creating shader control textures took 904ms
2023-10-24T14:31:44 38.602 INF Loading and parsing of generator took 706ms
2023-10-24T14:31:44 38.663 INF Computed 401 chunk groups containing a total of 7813 chunks. Largest group contains 462 chunks.
Exception: Incorrect region file header! /home/sdtdserver/.local/share/7DaysToDie/Saves/Navezgane/My Game/Region/r.2.4.7rg
  at RegionFile.Get (System.String dir, System.Int32 rX, System.Int32 rZ) [0x00060] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at RegionFileAccessMultipleChunks.GetRFC (System.Int32 regionX, System.Int32 regionZ, System.String _dir) [0x0005c] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at RegionFileAccessMultipleChunks.ReadDirectory (System.String _dir, System.Action`2[T1,T2] _chunkAndTimeStampHandler) [0x00075] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at RegionFileManager..ctor (System.String _loadDirectory, System.String _saveDirectory, System.Int32 _maxChunksInCache, System.Boolean _bSaveOnChunkDrop) [0x001f4] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at ChunkProviderGenerateWorldFromRaw+<Init>d__16.MoveNext () [0x00998] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00026] in <799d47ea3fd1460380efad1aacedaf7c>:0

ERROR: Shader UI/Default shader is not supported on this GPU (none of subshaders/fallbacks are suitable)

 

J'ai essayé de supprimer le fichier de région qui est incorrect, mais pas mieux... 
J'ai aussi essayé de récupérer le fichier avec un autre serveur mais pas mieux non plus.. 

2023-10-24T14:17:49 39.372 INF Loading and parsing of generator took 712ms
2023-10-24T14:17:49 39.439 INF Computed 401 chunk groups containing a total of 7813 chunks. Largest group contains 462 chunks.
2023-10-24T14:17:49 39.441 INF Started thread SaveChunks /home/sdtdserver/.local/share/7DaysToDie/Saves/Navezgane/My Game/Region
EndOfStreamException: Attempted to read past the end of the stream.
  at PooledBinaryReader.FillBuffer (System.Int32 _numBytes) [0x00030] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at PooledBinaryReader.ReadUInt32 () [0x00000] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at DecoObject.Read (System.IO.BinaryReader _br) [0x0001d] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at DecoManager.Read (System.IO.BinaryReader _br, System.Int32 _version, System.Boolean _resetExisting) [0x0001f] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at DecoManager.TryLoad () [0x00051] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at DecoManager+<OnWorldLoaded>d__33.MoveNext () [0x0024a] in <e9d9eff4796842fd937f9d9e966aa835>:0
  at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00026] in <799d47ea3fd1460380efad1aacedaf7c>:0

ERROR: Shader UI/Default shader is not supported on this GPU (none of subshaders/fallbacks are suitable)

Est-il possible de m'aider ? 
l’idée est de récupérer les données pour ne pas tout recommencer.

Merci d'avance pour votre aide.

 

PS : Désolé pour mon anglais, je suis français ^^

 

 

Edited October 24, 2023 by DarkEnd (see edit history)

[meganoth]

Deleting regions by hand should not be done anymore in A21 and newer. For that a new command was added in the console called "regionreset". Tryp "help regionreset" to get more info.

 

So if you can get the server running again you could try to use regionreset and check the logfile afterwards. But it is entirely possible that this isn't the only place your savegame got corrupted.

 

I assume you have free space again on your harddisk, right? If above doesn't work I would suggest installing a savegame-backup from a time when your harddisk still had free space left.

 

 

[DarkEnd]

Hello, 

Thanks for the answer.

 

I can't access to the console because the server did not start.

And the problem is : the only backup i have, is the backup not fully ...

Why, during the backup, the files would be corrupt ? i launch a simple save, so i imagine that it's a simple copy of files (not a replace).

 

I extend my FS to have free space.

Edited October 24, 2023 by DarkEnd (see edit history)

[meganoth]

I would assume the following happened: When the backup was made your harddisk ran out of space, so the backup was incomplete. Then you automatically started the server and when the server tried to add information (probably because of regular housekeeping tasks the game does when it starts up again, or a spawned zombie that destroyed a block, or ...) it couldn't and corrupted the savegame. Just a guess though

 

Please post the whole logfile to pastebin.com and post a link to it here. Usually a server will start even though a region file is corrupted

 

@Jugginator See above, there may be another error that prevents startup of this server, but generally, is it possible for the server to not start anymore even though only region files are corrupted? In that case the "regionreset" would be of no help

 

 

 

 

Edited October 24, 2023 by meganoth (see edit history)

[DarkEnd]

I don't find the whole logfile in my server.

But see logs in link : https://we.tl/t-xtPDfG3Rdz (wetransfer link)
 

[DarkEnd]

If i keep the region file : https://pastebin.com/raw/avc4hHcX

 

If i delete the region file : https://pastebin.com/raw/73qxrXP4

[DarkEnd]

I managed to access the console by making a backup of my server on my Windows PC.

but, i can't reset my region with the regionreset command.. see capture.

https://ibb.co/Mnn4j29

 

[Jugginator]

2 hours ago, meganoth said:

See above, there may be another error that prevents startup of this server, but generally, is it possible for the server to not start anymore even though only region files are corrupted? In that case the "regionreset" would be of no help

 

If the server is sort of running, the command in theory would still work. Mostly just iterating that deleting region files isn't supported anymore and issues are expected of it. If there's no other option sure.

I'll have to look into if it's possible to reset corrupted regions like you're doing. 

 

Honestly, DarkEnd, it looks like you have a large amount of corruption. I'm guessing what Meganoth said is what happened, when your drive got full some regions weren't able to be saved properly and then the backup also failed. For what it's worth, it's never okay to have a drive that's full, you want SSD's at least 35% free and HDD's 50%, otherwise you start hitting heavy performance issues quickly and can start limiting the lifetime of your SSD in that case.

 

In this case, I would try deleting region files. Do this until you can get back into the game, and then use the regionreset command to properly rebuild the regions. Hopefully that will prevent issues (regions are built by various systems touching save files, the command rebuilds them using those systems).