Just to close the circle on this, I found the problem.
After having increased the rmem_max I was no longer seeing any UDP packet drops. I was still not seeing some log messages, and at this point it was a configuration issue.
I noticed I had filters such as:
filter f_info { level(info);}
Obviously this was only catching info level messages. Changed to:
filter f_info {level(info...emerg);}
Sorry for the noise, but I still wanted to bring some closure in case someone runs into this. The initial packet drop blinded me into looking into some basic configs, thanks all for the suggestions!
> From: syslog-ng-request@lists.balabit.hu
> Subject: syslog-ng Digest, Vol 108, Issue 27
> To: syslog-ng@lists.balabit.hu
> Date: Wed, 30 Apr 2014 12:00:02 +0200
>
> Send syslog-ng mailing list submissions to
> syslog-ng@lists.balabit.hu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> or, via email, send a message with subject or body 'help' to
> syslog-ng-request@lists.balabit.hu
>
> You can reach the person managing the list at
> syslog-ng-owner@lists.balabit.hu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of syslog-ng digest..."
>
>
> Today's Topics:
>
> 1. Re: Ubuntu Precise -ng filling out buffer, dropping messages
> (Evan Rempel)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 29 Apr 2014 12:38:52 -0700
> From: Evan Rempel <erempel@uvic.ca>
> Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer,
> dropping messages
> To: syslog-ng@lists.balabit.hu
> Message-ID: <535FFFCC.8000203@uvic.ca>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> On 04/29/2014 12:20 PM, Chaman Chakalaka wrote:
> > I fixed the issue with udp being dropped at the system level by changing the linux core files but this time restarting the system, now I know they are missing somewhere between getting to the system and syslog-ng :(
> >
> > I'm lost once again...
> >
> > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> trysource s_network_udp { udp(so_rcvbuf(33554432) log_fetch_limit(20000) log_iw_size(1000000) ); };
>
>
> This assumes that you have a large net.core.rmem_max
>
> net.core.rmem_max = 52428800
>
> See how that goes.
>
>
> ------------------------------
>
> _______________________________________________
> syslog-ng maillist - syslog-ng@lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>
>
> End of syslog-ng Digest, Vol 108, Issue 27
> ******************************************