getHLdata and getCAdata for one species

[hgerritsen]

Hi, this is more an issue for the webservices but i think it would be useful to have an option to only extract HL and CA data for a single species. Currently you have to download the data from an entire survey, which seems a bit over-kill.
While we are at it, it might also be nice to be able to download multiple years, quarters or even surveys in one go. Something like this:

getHLdata(survey=c('EVHOE','IE-IGFS'),year=2003:2019,quarter=1:4,ahpia=126555)

Hans

[einarhjorleifsson]

Is this what you want on the menu?

remotes::install_github("einarhjorleifsson/obus@2a6c1f64ce0fda8c0167888488ae129b29e1e0f6")
tictoc::tic()
obus::dr_con("HL", trim = FALSE) |> 
  # this bug needs to be fixed
  dplyr::mutate(Year = as.integer(Year)) |>  
  dplyr::filter(Survey %in% c("EVHOE", "IE-IGFS"),
                Year %in% 2003:2019,
                latin == "Lophius piscatorius") |> 
  dplyr::collect() |>
  dplyr::glimpse()
tictoc::toc()

Returns 9205 records and 35 variables in less than a second! Think ICES seq is going this way via the IMBUS project.

> tictoc::tic()
> obus::dr_con("HL", trim = FALSE) |> 
+   # this bug needs to be fixed
+   dplyr::mutate(Year = as.integer(Year)) |>  
+   dplyr::filter(Survey %in% c("EVHOE", "IE-IGFS"),
+                 Year %in% 2003:2019,
+                 latin == "Lophius piscatorius") |> 
+   dplyr::collect() |>
+   dplyr::glimpse()
Rows: 9,205          
Columns: 35
$ RecordHeader          <chr> "HL", "HL", "HL", "HL", "HL", "HL", "HL", "HL", "HL", "HL", "HL", "HL", "HL", "HL", "…
$ Survey                <chr> "EVHOE", "EVHOE", "EVHOE", "EVHOE", "EVHOE", "EVHOE", "EVHOE", "EVHOE", "EVHOE", "EVH…
$ Quarter               <int> 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4…
$ Country               <chr> "FR", "FR", "FR", "FR", "FR", "FR", "FR", "FR", "FR", "FR", "FR", "FR", "FR", "FR", "…
$ Platform              <chr> "35HT", "35HT", "35HT", "35HT", "35HT", "35HT", "35HT", "35HT", "35HT", "35HT", "35HT…
$ Gear                  <chr> "GOV", "GOV", "GOV", "GOV", "GOV", "GOV", "GOV", "GOV", "GOV", "GOV", "GOV", "GOV", "…
$ SweepLength           <int> 100, 100, 100, 50, 50, 50, 100, 50, 50, 50, 50, 50, 50, 50, 50, 100, 100, 100, 100, 1…
$ GearExceptions        <chr> "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", "D", …
$ DoorType              <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
$ StationName           <chr> "H0465", "H0457", "H0457", "H0439", "H0452", "H0448", "H0432", "H0527", "H0527", "H05…
$ HaulNumber            <int> 64, 59, 59, 44, 57, 53, 37, 126, 126, 126, 126, 126, 126, 40, 126, 108, 157, 157, 110…
$ Year                  <int> 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2003, 2…
$ SpeciesCodeType       <chr> "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", "W", …
$ SpeciesCode           <chr> "126555", "126555", "126555", "126555", "126555", "126555", "126555", "126555", "1265…
$ SpeciesValidity       <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", …
$ SpeciesSex            <chr> "F", "M", "F", "F", "F", "F", "M", "M", "M", "M", "F", "F", "F", "M", "U", "U", "M", …
$ TotalNumber           <dbl> 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 1, 1, 1, 2, 2, 3, 3, 3, 1, 3, 3, 3, 1, 1, 1, 1…
$ SpeciesCategory       <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ SubsampledNumber      <int> 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 1, 1, 1, 2, 2, 3, 3, 3, 1, 3, 3, 3, 1, 1, 1, 1…
$ SubsamplingFactor     <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ SubsampleWeight       <int> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
$ SpeciesCategoryWeight <int> 1290, 1270, 890, 840, 360, 600, 2100, 4260, 4260, 4260, 4840, 4840, 4840, 430, 10, 70…
$ LengthCode            <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", …
$ LengthClass           <int> 40, 47, 39, 35, 28, 34, 51, 43, 46, 53, 21, 35, 64, 33, 7, 16, 54, 55, 24, 29, 30, 26…
$ NumberAtLength        <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ DevelopmentStage      <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
$ LengthType            <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
$ ValidAphiaID          <int> 126555, 126555, 126555, 126555, 126555, 126555, 126555, 126555, 126555, 126555, 12655…
$ latin                 <chr> "Lophius piscatorius", "Lophius piscatorius", "Lophius piscatorius", "Lophius piscato…
$ DateofCalculation     <int> 20140402, 20140402, 20140402, 20140402, 20140402, 20140402, 20140402, 20140402, 20140…
$ .id                   <chr> "EVHOE:2003:4:FR:35HT:GOV:H0465:64", "EVHOE:2003:4:FR:35HT:GOV:H0457:59", "EVHOE:2003…
$ length_cm             <dbl> 40, 47, 39, 35, 28, 34, 51, 43, 46, 53, 21, 35, 64, 33, 7, 16, 54, 55, 24, 29, 30, 26…
$ length_mm             <dbl> 400, 470, 390, 350, 280, 340, 510, 430, 460, 530, 210, 350, 640, 330, 70, 160, 540, 5…
$ n_haul                <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
$ n_hour                <dbl> 2.000000, 2.000000, 2.000000, 2.068966, 2.000000, 2.000000, 2.000000, 2.000000, 2.000…
> tictoc::toc()
0.489 sec elapsed

[hgerritsen]

Yes, that would be perfect. If the IMBUS project is going to implement it, the issue can be closed.