Skip to content

Conversation

@David-McKenna
Copy link
Contributor

Hey Dave,

We've been looking to use FIFOs as an input to rawspec at the LOFAR stations given we're performing a packet-to-GUPPI raw conversion, so preventing a write to disk would save a good chunk of time while processing data.

Due to the use of seeks (via lseek) in rawspec, it can't handle FIFOs as an input in the current version. This PR introduces an alternative code path where rawspec will detect if the input is a FIFO or PIPE, and now either

  • Read only the data required from FIFOs, rather than reading and reverse seeking (in rawspec_raw_read_header)
  • Read data from FIFOs into a discard buffer to seek into the future (for handling schan, in main)

To achieve the latter, I've created a new function, read_to_skip based on calls to read_fully, this has an 8k buffer that it repeatedly reads data into until the requested data has been consumed. I tried benchmarking a few options for the output array (the array of constant length, a variable length array and a malloc'd array) and found the fixed 8k array was the best option (VLA was faster in some cases, but segfaulted for larger read lengths).

I've tried to follow your coding style and approach while implementing this, but let me know if you want anything changed or if there's way you feel I could improve the implementation.

Cheers,
David

@David-McKenna David-McKenna changed the base branch from master to v2.3-branch July 7, 2021 10:29
@texadactyl texadactyl marked this pull request as draft November 5, 2021 18:43
@texadactyl
Copy link
Contributor

@David-McKenna
This PR is pending your analysis of the hashpipe technique - direct call vs using a pipe.

cc: @david-macmahon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants