Browsing logs

[siilike]

It would be great to be able to browse logs:

1. Filter out the initial results (e.g. a particular application on a particular day)
2. Find something from the results (e.g. by user ID)
3. View the previous or next log entries (which may or may not contain the user ID) which would be loaded as needed (pages)

Basically, like less is used on log files.

Currently, the only way to achieve this would be to load the whole log into the browser, but this has its limitations.

[HumanUser]

Hello, thanks for your request.

Am I correct in understanding that you want functionality similar to grep with the -A and -B flags for text files?

$ grep -n 'The above copyright notice a' LICENSE
12:The above copyright notice and this permission notice shall be included in all

$ grep -n -A 5 -B 5 'The above copyright notice a' LICENSE
7-in the Software without restriction, including without limitation the rights
8-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9-copies of the Software, and to permit persons to whom the Software is
10-furnished to do so, subject to the following conditions:
11-
12:The above copyright notice and this permission notice shall be included in all
13-copies or substantial portions of the Software.
14-
15-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

[siilike]

That would also be useful, but not exactly.

First, a base result set needs to be fetched, let's say it's the application log from today for a certain client, 500k rows in total.

Next, from that base result set, search for all entries that have a certain user ID, let's say there are 50 that are displayed in a table.

Next, once we find an interesting entry, click a button to show 25 previous log entries i.e. that do not have the user ID, but are still part of the base result set. If that wasn't enough, show 25 more. Or 25 next ones.

Finally, going down the rabbit hole, we might want to filter out all log entries with a certain request ID i.e. take the original interesting entry what has a certain request ID, and then show all previous log entries with the same request ID (on busy sites the immediately previous ones might just be noise).

So, basically an easy way to search for something that can only be filtered out approximately.

Currently, I am using less for this, for example:

1. open the relevant daily log file (or grep the relevant parts)
2. go to the end of the file
3. reverse search the user ID
4. check a few pages upwards
5. take the next result
6. once something interesting is found, maybe reverse search the request ID, or grep all entries with it

Telescope is already very nice, but I can see many of those "it's faster to just use less" cases.

[HumanUser]

Thank you for the clarification.

I now have a better understanding of what needs to be implemented. However, I can't promise that this functionality can be developed quickly.

When it comes to less/grep, I myself am a strong advocate of the idea that there's usually nothing better for searching and working with text files. However, modern infrastructure environments are increasingly limiting direct access to text files, especially in the context of Kubernetes and cloud-based setups.

[soda-pop-ice-cream]

Maybe @siilike waned something like this, when you found event that you need, you click on it's time and see menu like this:

[ansgri]

Opensearch Dashboards has a similar feature, 'view surrounding documents', it works exactly like -C 5 in grep. It would probably be easiest to just include a button that sets the context filter in the filter expression, but it doesn't seem to be expressible in FlyQL. As a close second, adding a 'range' button to datetime and numeric fields (next to existing = and !=) would be very helpful too.