[syslog-ng] Processing orphaned disk queue files

[László Várady]

Hi,

You are probably looking for the `dqtool assign` command in syslog-ng OSE.
Please note that this command has a bug that makes it inoperable, so you 
can only use it properly if you upgrade to the upcoming syslog-ng 4.1.0 
release.

Let me share a quick tutorial.

1. Prepare a destination with the appropriate disk-buffer type 
(reliable/non-reliable):

destination {
     network("localhost" port(4444) disk-buffer(reliable(yes) 
disk-buf-size(10G)));
     http(url("https://test.tld/logs") workers(3) 
disk-buffer(reliable(yes) disk-buf-size(10G)));
};

2. Start and stop syslog-ng to generate persist entries for the your 
destination.

$ bin/persist-tool dump var/syslog-ng.persist

http(https://test.tld/logs,).1.queue = { "value": "..." }
http(https://test.tld/logs,).2.queue = { "value": "..." }
http(https://test.tld/logs,) = { "value": "..." }
afsocket_dd_qfile(stream,localhost:4444) = { "value": "..." }

3. Find the name of the destination into which you want to inject the 
orphaned queue file. For example: "afsocket_dd_qfile(stream,localhost:4444)"

4. Inject the queue file:

$ bin/dqtool assign -p var/syslog-ng.persist -n 
"afsocket_dd_qfile(stream,localhost:4444)" orphaned/syslog-ng-00001.rqf

You can now start syslog-ng and monitor its "queued" counters with 
`syslog-ng-ctl stats` to make sure the queue file is drained.

----

Alternatively, you can use a source called "example-diskq-source", if 
you are okay with trying experimental features.
In that case, you need to remove your disk-buffer declaration from your 
destination, and declare the diskq-source right before
the destionation:

source {
     example-diskq-source(file("/tmp/syslog-ng-00000.rqf"));
};
destination {
     network("localhost" port(4444));
};

----

The third and the less user-friendly alternative is using `persist-tool 
dump`:

$ bin/persist-tool dump var/syslog-ng.persist > dump.txt

You can edit the dump file manually (the hex values represent the 
disk-buffer file name in ASCII, you can replace them),
and then you can reimport the edited dump:

$ bin/persist-tool add dump.txt -o var/

--
László Várady

On 2023. 03. 06. 18:31, Steve Bernacki wrote:
> Hello,
>
> I am using syslog-ng OSE 3.29. I have two destinations that I recently 
> redefined in order to spread load out across a number of receivers. 
> However, having done this, I now have orphaned queue files of 
> considerable size that I'd like to process through.
>
> I found one reference on how to do this with syslog-ng PE which is 
> documented at 
> https://support-public.cfm.quest.com/46246_syslog-ng-pe-diskbuffer-recovery.pdf, 
> but I'm finding the OSE tools are acting slightly differently, for 
> example persist-tool dump prints its output as hex instead of ASCII, 
> as is documented in the PDF I linked to above. In addition, the 
> process is fairly involved, which makes me nervous to experiment with 
> this.
>
> Looking at the command output for "dqtool --help", I see there are a 
> number of commands listed that are not documented anywhere, like 
> "relocate" and "assign", which look like they may be helpful to me. 
> Does anyone know if these can be used to help me process through these 
> logs?
>
> Thanks
> Steve
>
>
> ______________________________________________________________________________ 
>
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: 
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>