Error running pop calibration notebook

[bpstewar]

When running the gep-onsset code, I get the following error:

For Example,

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\\_libs\\hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\\_libs\\hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'ElecPop'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[44], line 1
----> 1 pop_modelled, urban_modelled = onsseter.calibrate_current_pop_and_urban(pop_start_year, urban_ratio_start_year)
      3 onsseter.project_pop_and_urban(pop_modelled, end_year_pop, urban_modelled, urban_ratio_end_year, start_year, end_year, intermediate_year)
      5 display(Markdown('#### The csv file has been imported correctly. Here is a preview:'))

File ~\AppData\Local\anaconda3\Lib\site-packages\onsset\onsset.py:975, in SettlementProcessor.calibrate_current_pop_and_urban(self, pop_actual, urban_current)
    971 print('The calibrated population differs by {:.2f}. '
    972       'In case this is not acceptable please revise this part of the code'.format(pop_diff))
    974 # TODO Why do we apply the ratio to elec_pop? Shouldn't the calibration take place before defining elec_pop?
--> 975 self.df[SET_ELEC_POP_CALIB] = self.df[SET_ELEC_POP] * pop_ratio
    977 logging.info('Urban/rural calibration process')
    978 # TODO As indicated below, HRSL classifies in 0, 1 and 2; I don't get why if statement uses 3 here.

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\frame.py:4102, in DataFrame.__getitem__(self, key)
   4100 if self.columns.nlevels > 1:
   4101     return self._getitem_multilevel(key)
-> 4102 indexer = self.columns.get_loc(key)
   4103 if is_integer(indexer):
   4104     indexer = [indexer]

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3812, in Index.get_loc(self, key)
   3807     if isinstance(casted_key, slice) or (
   3808         isinstance(casted_key, abc.Iterable)
   3809         and any(isinstance(x, slice) for x in casted_key)
   3810     ):
   3811         raise InvalidIndexError(key)
-> 3812     raise KeyError(key) from err
   3813 except TypeError:
   3814     # If we have a listlike key, _check_indexing_error will raise
   3815     #  InvalidIndexError. Otherwise we fall through and re-raise
   3816     #  the TypeError.
   3817     self._check_indexing_error(key)

KeyError: 'ElecPop'

image.png
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\\_libs\\hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\\_libs\\hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'GridCellArea'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[45], line 1
----> 1 onsseter.condition_df()
      2 onsseter.df['GridPenalty'] = onsseter.grid_penalties(onsseter.df)
      3 onsseter.df['WindCF'] = onsseter.calc_wind_cfs()

File ~\AppData\Local\anaconda3\Lib\site-packages\onsset\onsset.py:735, in SettlementProcessor.condition_df(self)
    733 self.df[SET_NIGHT_LIGHTS] = pd.to_numeric(self.df[SET_NIGHT_LIGHTS], errors='coerce')
    734 self.df[SET_POP] = pd.to_numeric(self.df[SET_POP], errors='coerce')
--> 735 self.df[SET_GRID_CELL_AREA] = pd.to_numeric(self.df[SET_GRID_CELL_AREA], errors='coerce')
    736 self.df[SET_ELEC_POP] = pd.to_numeric(self.df[SET_ELEC_POP], errors='coerce')
    737 self.df[SET_GHI] = pd.to_numeric(self.df[SET_GHI], errors='coerce')

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\frame.py:4102, in DataFrame.__getitem__(self, key)
   4100 if self.columns.nlevels > 1:
   4101     return self._getitem_multilevel(key)
-> 4102 indexer = self.columns.get_loc(key)
   4103 if is_integer(indexer):
   4104     indexer = [indexer]

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3812, in Index.get_loc(self, key)
   3807     if isinstance(casted_key, slice) or (
   3808         isinstance(casted_key, abc.Iterable)
   3809         and any(isinstance(x, slice) for x in casted_key)
   3810     ):
   3811         raise InvalidIndexError(key)
-> 3812     raise KeyError(key) from err
   3813 except TypeError:
   3814     # If we have a listlike key, _check_indexing_error will raise
   3815     #  InvalidIndexError. Otherwise we fall through and re-raise
   3816     #  the TypeError.
   3817     self._check_indexing_error(key)

KeyError: 'GridCellArea'

Please kindly let me know how to solve the errors.

I downloaded and saved gep-onsset-master from the following link.

https://github.com/global-electrification-platform/gep-onsset.git

[KYOJIFUJII]

Hi bpstewart,

Thanks for opening a session on the gep-onsset.
Here is the csv file that I used as an input to gep-onsset.
BeninCalibrated.csv

Other errors that I got in the following process are as follows.

KeyError Traceback (most recent call last)
File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
3804 try:
-> 3805 return self._engine.get_loc(casted_key)
3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\_libs\hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\_libs\hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'TransformerDist'

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
Cell In[21], line 8
5 max_mv_line_distance = 2
6 max_hv_line_distance = 5
----> 8 elec_modelled, urban_internal_elec_ratio, rural_internal_elec_ratio = onsseter.elec_current_and_future(elec_ratio_start_year,
9 urban_elec_ratio,
10 rural_elec_ratio,
11 start_year,
12 min_night_lights=min_night_lights,
13 min_pop=min_pop,
14 max_transformer_dist=max_service_transformer_distance,
15 max_mv_dist=max_mv_line_distance,
16 max_hv_dist=max_hv_line_distance)

File ~\AppData\Local\anaconda3\Lib\site-packages\onsset\onsset.py:1094, in SettlementProcessor.elec_current_and_future(self, elec_actual, elec_actual_urban, elec_actual_rural, start_year, min_night_lights, min_pop, max_transformer_dist, max_mv_dist, max_hv_dist)
1090 self.df[SET_ELEC_CURRENT] = 0
1092 # This if function here skims through T&D columns to identify if any non 0 values exist;
1093 # Then it defines calibration method accordingly.
-> 1094 if max(self.df[SET_DIST_TO_TRANS]) > 0:
1095 self.df[SET_CALIB_GRID_DIST] = self.df[SET_DIST_TO_TRANS]
1096 priority = 1

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\frame.py:4102, in DataFrame.getitem(self, key)
4100 if self.columns.nlevels > 1:
4101 return self._getitem_multilevel(key)
-> 4102 indexer = self.columns.get_loc(key)
4103 if is_integer(indexer):
4104 indexer = [indexer]

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3812, in Index.get_loc(self, key)
3807 if isinstance(casted_key, slice) or (
3808 isinstance(casted_key, abc.Iterable)
3809 and any(isinstance(x, slice) for x in casted_key)
3810 ):
3811 raise InvalidIndexError(key)
-> 3812 raise KeyError(key) from err
3813 except TypeError:
3814 # If we have a listlike key, _check_indexing_error will raise
3815 # InvalidIndexError. Otherwise we fall through and re-raise
3816 # the TypeError.
3817 self._check_indexing_error(key)

KeyError: 'TransformerDist'

---

KeyError Traceback (most recent call last)
File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
3804 try:
-> 3805 return self._engine.get_loc(casted_key)
3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\_libs\hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\_libs\hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'CurrentHVLineDist'

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
Cell In[27], line 1
----> 1 onsseter.current_mv_line_dist()
3 try:
4 onsseter.df.reset_index(inplace=True)

File ~\AppData\Local\anaconda3\Lib\site-packages\onsset\onsset.py:1313, in SettlementProcessor.current_mv_line_dist(self)
1311 logging.info('Determine current MV line length')
1312 self.df[SET_MV_CONNECT_DIST] = 0
-> 1313 self.df.loc[self.df[SET_ELEC_CURRENT] == 1, SET_MV_CONNECT_DIST] = self.df[SET_HV_DIST_CURRENT]
1314 self.df[SET_MIN_TD_DIST] = self.df[[SET_MV_DIST_PLANNED, SET_HV_DIST_PLANNED]].min(axis=1)

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\frame.py:4102, in DataFrame.getitem(self, key)
4100 if self.columns.nlevels > 1:
4101 return self._getitem_multilevel(key)
-> 4102 indexer = self.columns.get_loc(key)
4103 if is_integer(indexer):
4104 indexer = [indexer]

File ~\AppData\Local\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3812, in Index.get_loc(self, key)
3807 if isinstance(casted_key, slice) or (
3808 isinstance(casted_key, abc.Iterable)
3809 and any(isinstance(x, slice) for x in casted_key)
3810 ):
3811 raise InvalidIndexError(key)
-> 3812 raise KeyError(key) from err
3813 except TypeError:
3814 # If we have a listlike key, _check_indexing_error will raise
3815 # InvalidIndexError. Otherwise we fall through and re-raise
3816 # the TypeError.
3817 self._check_indexing_error(key)

KeyError: 'CurrentHVLineDist'

[KYOJIFUJII]

@bpstewar
Will you invite Mr. Andreas Sahlberg and Mr. Alexandros Korkovelos to this chat?

[KYOJIFUJII]

I could finally run the GEP-generator without any errors, by installing gep-onsset-1.1 and QGIS 3.10.7.
Other combinations do not work.