since yesterday I’ve got really strange behavior, my gridnetCore dies after few hours ( is closed unexpectedly) , I’m not able to find the reason in logs. Could you help?
Block cache LRUCache@0000023EC859D8B0#29888 capacity: 8.00 MB collections: 22 last_copies: 4 last_secs: 0.000249 secs_since: 12
Block cache entry stats(count,size,portion): DataBlock(1642,6.32 MB,79.0069%) Misc(1,0.00 KB,0%)
** File Read Latency Histogram By Level [default] **
2023/11/14-23:15:16.071607 76e4 [db\db_impl\db_impl_write.cc:1817] [default] New memtable created with log file: #1838. Immutable memtables: 0.
2023/11/14-23:15:16.071765 36b4 (Original Log Time 2023/11/14-23:15:16.071755) [db\db_impl\db_impl_compaction_flush.cc:2701] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
2023/11/14-23:15:16.071776 36b4 [db\flush_job.cc:819] [default] [JOB 19] Flushing memtable with next log file: 1838
2023/11/14-23:15:16.071794 36b4 EVENT_LOG_v1 {"time_micros": 1700000116071784, "job": 19, "event": "flush_started", "num_memtables": 1, "num_entries": 1355, "num_deletes": 20, "total_data_size": 68305774, "memory_usage": 68327896, "flush_reason": "Write Buffer Full"}
2023/11/14-23:15:16.071797 36b4 [db\flush_job.cc:848] [default] [JOB 19] Level-0 flush table #1839: started
2023/11/14-23:15:16.109606 36b4 EVENT_LOG_v1 {"time_micros": 1700000116109576, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 1839, "file_size": 24744571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 24742331, "index_size": 1282, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 0, "raw_key_size": 35755, "raw_average_key_size": 46, "raw_value_size": 24707762, "raw_average_value_size": 32046, "num_data_blocks": 64, "num_entries": 771, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; ", "creation_time": 1699999254, "oldest_key_time": 1699999254, "file_creation_time": 1700000116, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff1a07f2-7d3c-11ee-b272-047c16719481", "db_session_id": "MG6R0DQ2Q36STH8A1HQE", "orig_file_number": 1839}}
2023/11/14-23:15:16.109617 36b4 [db\flush_job.cc:937] [default] [JOB 19] Level-0 flush table #1839: 24744571 bytes OK
2023/11/14-23:15:16.109628 36b4 [db\flush_job.cc:986] [default] [JOB 19] Flush lasted 37864 microseconds, and 0 cpu microseconds.
2023/11/14-23:15:16.110697 36b4 (Original Log Time 2023/11/14-23:15:16.109632) [db\memtable_list.cc:471] [default] Level-0 commit table #1839 started
2023/11/14-23:15:16.110705 36b4 (Original Log Time 2023/11/14-23:15:16.110598) [db\memtable_list.cc:675] [default] Level-0 commit table #1839: memtable #1 done
2023/11/14-23:15:16.110708 36b4 (Original Log Time 2023/11/14-23:15:16.110619) EVENT_LOG_v1 {"time_micros": 1700000116110610, "job": 19, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 3, 4, 0, 0, 0, 0], "immutable_memtables": 0}
2023/11/14-23:15:16.110711 36b4 (Original Log Time 2023/11/14-23:15:16.110648) [db\db_impl\db_impl_compaction_flush.cc:264] [default] Level summary: files[4 3 4 0 0 0 0] max score 1.00
2023/11/14-23:15:16.110720 36b4 [db\db_impl\db_impl_files.cc:439] [JOB 19] Try to delete WAL files size 68312529, prev total WAL file size 69422969, number of live WAL files 2.
2023/11/14-23:15:16.110725 4ecc [db\compaction\compaction_job.cc:2334] [default] [JOB 20] Compacting 4@0 + 3@1 files to L1, score 1.00
2023/11/14-23:15:16.110737 4ecc [db\compaction\compaction_job.cc:2338] [default] Compaction start summary: Base version 20 Base level 0, inputs: [1839(23MB) 1837(23MB) 1835(23MB) 1833(23MB)], [1829(64MB) 1830(64MB) 1831(65MB)]
2023/11/14-23:15:16.110753 4ecc EVENT_LOG_v1 {"time_micros": 1700000116110742, "job": 20, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [1839, 1837, 1835, 1833], "files_L1": [1829, 1830, 1831], "score": 1, "input_data_size": 302810094}
2023/11/14-23:15:16.122555 36b4 [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001836.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.316275 4ecc [db\compaction\compaction_job.cc:1942] [default] [JOB 20] Generated table #1840: 280165 keys, 67379051 bytes
2023/11/14-23:15:16.316325 4ecc EVENT_LOG_v1 {"time_micros": 1700000116316293, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 1840, "file_size": 67379051, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 67112233, "index_size": 265860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 0, "raw_key_size": 12867296, "raw_average_key_size": 45, "raw_value_size": 54905581, "raw_average_value_size": 195, "num_data_blocks": 13285, "num_entries": 280165, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; ", "creation_time": 1699341174, "oldest_key_time": 0, "file_creation_time": 1700000116, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff1a07f2-7d3c-11ee-b272-047c16719481", "db_session_id": "MG6R0DQ2Q36STH8A1HQE", "orig_file_number": 1840}}
2023/11/14-23:15:16.494083 4ecc [db\compaction\compaction_job.cc:1942] [default] [JOB 20] Generated table #1841: 177543 keys, 67477075 bytes
2023/11/14-23:15:16.494131 4ecc EVENT_LOG_v1 {"time_micros": 1700000116494100, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 1841, "file_size": 67477075, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 67112538, "index_size": 363578, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 0, "raw_key_size": 8344521, "raw_average_key_size": 47, "raw_value_size": 59311753, "raw_average_value_size": 334, "num_data_blocks": 16653, "num_entries": 177543, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; ", "creation_time": 1699341174, "oldest_key_time": 0, "file_creation_time": 1700000116, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff1a07f2-7d3c-11ee-b272-047c16719481", "db_session_id": "MG6R0DQ2Q36STH8A1HQE", "orig_file_number": 1841}}
2023/11/14-23:15:16.674448 4ecc [db\compaction\compaction_job.cc:1942] [default] [JOB 20] Generated table #1842: 151722 keys, 70062128 bytes
2023/11/14-23:15:16.674498 4ecc EVENT_LOG_v1 {"time_micros": 1700000116674467, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 1842, "file_size": 70062128, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 69748682, "index_size": 312488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 0, "raw_key_size": 7130868, "raw_average_key_size": 46, "raw_value_size": 63094963, "raw_average_value_size": 415, "num_data_blocks": 14261, "num_entries": 151722, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; ", "creation_time": 1699341174, "oldest_key_time": 0, "file_creation_time": 1700000116, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff1a07f2-7d3c-11ee-b272-047c16719481", "db_session_id": "MG6R0DQ2Q36STH8A1HQE", "orig_file_number": 1842}}
2023/11/14-23:15:16.678945 4ecc [db\compaction\compaction_job.cc:2002] [default] [JOB 20] Compacted 4@0 + 3@1 files to L1 => 204918254 bytes
2023/11/14-23:15:16.679966 4ecc (Original Log Time 2023/11/14-23:15:16.679899) [db\compaction\compaction_job.cc:962] [default] compacted to: files[0 3 4 0 0 0 0] max score 0.76, MB/sec: 537.1 rd, 363.4 wr, level 1, files in(4, 3) out(3 +0 blob) MB in(94.3, 194.5 +0.0 blob) out(195.4 +0.0 blob), read-write-amplify(5.1) write-amplify(2.1) OK, records in: 609513, records dropped: 83 output_compression: NoCompression
2023/11/14-23:15:16.679978 4ecc (Original Log Time 2023/11/14-23:15:16.679926) EVENT_LOG_v1 {"time_micros": 1700000116679908, "job": 20, "event": "compaction_finished", "compaction_time_micros": 563831, "compaction_time_cpu_micros": 0, "output_level": 1, "num_output_files": 3, "total_output_size": 204918254, "num_input_records": 609513, "num_output_records": 609430, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 4, 0, 0, 0, 0]}
2023/11/14-23:15:16.681470 4ecc [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001839.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.681483 4ecc EVENT_LOG_v1 {"time_micros": 1700000116681479, "job": 20, "event": "table_file_deletion", "file_number": 1839}
2023/11/14-23:15:16.683177 4ecc [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001837.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.683193 4ecc EVENT_LOG_v1 {"time_micros": 1700000116683187, "job": 20, "event": "table_file_deletion", "file_number": 1837}
2023/11/14-23:15:16.685071 4ecc [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001835.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.685084 4ecc EVENT_LOG_v1 {"time_micros": 1700000116685080, "job": 20, "event": "table_file_deletion", "file_number": 1835}
2023/11/14-23:15:16.686932 4ecc [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001833.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.686976 4ecc EVENT_LOG_v1 {"time_micros": 1700000116686972, "job": 20, "event": "table_file_deletion", "file_number": 1833}
2023/11/14-23:15:16.691755 4ecc [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001831.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.691767 4ecc EVENT_LOG_v1 {"time_micros": 1700000116691763, "job": 20, "event": "table_file_deletion", "file_number": 1831}
2023/11/14-23:15:16.696356 4ecc [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001830.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.696369 4ecc EVENT_LOG_v1 {"time_micros": 1700000116696365, "job": 20, "event": "table_file_deletion", "file_number": 1830}
2023/11/14-23:15:16.700162 4ecc [file\delete_scheduler.cc:77] Deleted file C:\Users\MM\AppData\Roaming\GRIDNET\TestNet-StateDB/001829.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2023/11/14-23:15:16.700172 4ecc EVENT_LOG_v1 {"time_micros": 1700000116700169, "job": 20, "event": "table_file_deletion", "file_number": 1829}
2023/11/14-23:22:49.607808 a48c [db\db_impl\db_impl.cc:1005] ------- DUMPING STATS -------
2023/11/14-23:22:49.607833 a48c [db\db_impl\db_impl.cc:1006]
** DB Stats **
Uptime(secs): 13212.3 total, 600.0 interval
Cumulative writes: 15K writes, 15K keys, 15K commit groups, 1.0 writes per commit group, ingest: 0.92 GB, 0.07 MB/s
Cumulative WAL: 15K writes, 0 syncs, 15861.00 writes per sync, written: 0.92 GB, 0.07 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 178 writes, 178 keys, 178 commit groups, 1.0 writes per commit group, ingest: 47.80 MB, 0.08 MB/s
Interval WAL: 178 writes, 0 syncs, 178.00 writes per sync, written: 0.05 GB, 0.08 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.3 0.3 0.0 1.0 0.0 571.4 0.58 0.00 14 0.041 0 0 0.0 0.0
L1 3/0 195.43 MB 0.8 1.2 0.4 0.8 0.8 0.0 0.0 2.3 516.2 360.8 2.33 0.00 4 0.583 2600K 605 0.0 0.0
L2 4/0 257.32 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Sum 7/0 452.74 MB 0.0 1.2 0.4 0.8 1.1 0.3 0.1 3.5 413.7 402.6 2.91 0.00 18 0.162 2600K 605 0.0 0.0
Int 0/0 0.00 KB 0.0 0.3 0.1 0.2 0.2 0.0 0.0 9.3 479.9 364.0 0.60 0.00 2 0.301 609K 83 0.0 0.0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Low 0/0 0.00 KB 0.0 1.2 0.4 0.8 0.8 0.0 0.0 0.0 516.2 360.8 2.33 0.00 4 0.583 2600K 605 0.0 0.0
High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.3 0.3 0.0 0.0 0.0 562.1 0.55 0.00 13 0.042 0 0 0.0 0.0
User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 729.7 0.03 0.00 1 0.032 0 0 0.0 0.0
Blob file count: 0, total size: 0.0 GB
Uptime(secs): 13212.3 total, 600.0 interval
Flush(GB): cumulative 0.323, interval 0.023
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 1.14 GB write, 0.09 MB/s write, 1.18 GB read, 0.09 MB/s read, 2.9 seconds
Interval compaction: 0.21 GB write, 0.37 MB/s write, 0.28 GB read, 0.48 MB/s read, 0.6 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
Block cache LRUCache@0000023EC859D8B0#29888 capacity: 8.00 MB collections: 23 last_copies: 4 last_secs: 0.000167 secs_since: 12
Block cache entry stats(count,size,portion): DataBlock(1304,5.04 MB,62.9667%) Misc(1,0.00 KB,0%)
** File Read Latency Histogram By Level [default] **
