killed when i try to push blocks into database geth/infura

[dirmansebastian]

Describe the bug
killed

To Reproduce
run ether_sql --settings='PersonalGethSettings' scrape_block_range --start_block_number 6305955 , my eth light geth node is syncing and it's over block 6305955

but, i get killed when i try to push data in database

i get killed too if i use same command with infura: ether_sql --settings='PersonalInfuraSettings' scrape_block_range --start_block_number 6305955

i've modified settings.py like that:

but, if i do ether_sql scrape_block --block_number 6305955 it also connect to DefaultSettings(infura, not my api ) and it works

Expected behavior
to not get killed and to push blocks into database

Desktop (please complete the following information):

• OS: Ubuntu 16.04
• Node Geth

[ankitchiplunkar]

my eth light geth node is syncing and it's over block 6305955

I have never tried pushing blocks using light geth blocks.
I will try it and update this issue.

but, if i do ether_sql scrape_block --block_number 6305955 it also connect to DefaultSettings(infura, not my api ) and it works

To use your personal infura setting use this command:
ether_sql --settings='PersonalInfuraSettings' scrape_block_range --start_block_number 6305955

PS: Thanks a lot for stres testing the library
PPS: refrain from posting your private infura key on a public forum like this.

[dirmansebastian]

that's how i've used it: ether_sql --settings='PersonalInfuraSettings' scrape_block_range --start_block_number 6307820, but no luck

anytime, i'm glad i can help a little
it's for test, no worry, if people are lazy to create an account on infura they can use it

[ankitchiplunkar]

Strange, I tried the command and it works for me:

$ ether_sql --settings=PersonalInfuraSettings scrape_block --block_number=6307820
[2018-09-10 22:14:30,229][INFO][ether_sql.session] Connected to the db ether_sql
[2018-09-10 22:14:30,231][INFO][ether_sql.session] Connected to Infura node
[2018-09-10 22:14:30,231][INFO][ether_sql.globals] Pushing the session PersonalInfuraSettings in local thread
[2018-09-10 22:15:01,746][INFO][ether_sql.tasks.scrapper] Commiting block: 6307820 to sql

More, stranger thing is that the above command works for you when using DefaultSettings but does not work with PersonalInfuraSettings

Can you add the line LOG_LEVEL = "DEBUG" in your class PersonalInfuraSettings to see where is the process actually being killed, and copy paste the output?

[dirmansebastian]

(envname) [20:45 ubuntu@ip-172-26-15-152 ether_sql] > ether_sql --settings='PersonalInfuraSettings' scrape_block_range --start_block_number 6307821
[2018-09-10 20:45:16,099][DEBUG][ether_sql.session] DEBUG
[2018-09-10 20:45:16,164][INFO][ether_sql.session] Connected to the db ether_sql
[2018-09-10 20:45:16,165][INFO][ether_sql.session] Connected to Infura node
[2018-09-10 20:45:16,165][INFO][ether_sql.globals] Pushing the session PersonalInfuraSettings in local thread
[2018-09-10 20:45:16,224][DEBUG][web3.RequestManager] Making request. Method: eth_blockNumber
[2018-09-10 20:45:16,225][DEBUG][web3.providers.HTTPProvider] Making request HTTP. URI: https://mainnet.infura.io/HcekpL4KhjR4L0pCa4o8, Method: eth_blockNumber
[2018-09-10 20:45:16,230][DEBUG][urllib3.connectionpool] Starting new HTTPS connection (1): mainnet.infura.io:443
[2018-09-10 20:45:16,617][DEBUG][urllib3.connectionpool] https://mainnet.infura.io:443 "POST /HcekpL4KhjR4L0pCa4o8 HTTP/1.1" 200 44
[2018-09-10 20:45:16,617][DEBUG][web3.providers.HTTPProvider] Getting response HTTP. URI: https://mainnet.infura.io/HcekpL4KhjR4L0pCa4o8, Method: eth_blockNumber, Response: {'jsonrpc': '2.0', 'id': 0, 'result': '0x604106'}
[2018-09-10 20:45:16,618][DEBUG][ether_sql.cli] 6308102
[2018-09-10 20:45:16,618][DEBUG][ether_sql.cli] 6307821
[2018-09-10 20:45:16,618][DEBUG][ether_sql.models.blocks] 6305957
Killed

[dirmansebastian]

Response: {'jsonrpc': '2.0', 'id': 0, 'result': '0x604106'}, there should be 'id':1 right?

[ankitchiplunkar]

I think I got the reason why the process is being killed. The command first checks for missing blocks in the sql i.e. number of blocks which are missing between start_block_number and block 0, this is a heavy task and might be the reason for Killedcommand.

[2018-09-10 20:45:16,618][DEBUG][ether_sql.models.blocks] 6305957

This log is returned at line below and the process is being killed after that.

ether_sql/ether_sql/models/blocks.py

Line 111 in 0ce3a74

logger.debug(max_block_number)

Try running the above command with setting --no-fill_gaps

$ ether_sql --settings=PersonalInfuraSettings scrape_block_range --start_block_number=6000000 --no-fill_gaps

Response: {'jsonrpc': '2.0', 'id': 0, 'result': '0x604106'}, there should be 'id':1 right?

Yes the id should be 1 but we are not using the id in any way it to kill our process.

[dirmansebastian]

yes, u're right, i got it now.
--no-fill_gaps work as workaround with infura

running same command with Geth: ether_sql --settings='PersonalGethSettings' scrape_block_range --start_block_number 6308345 --no-fill_gaps

(envname) [21:50 ubuntu@ip-172-26-15-152 ether_sql] > ether_sql --settings=PersonalGethSettings scrape_block_range --start_block_number=6000048 --no-fill_gaps
[2018-09-10 21:51:13,389][INFO][ether_sql.session] Connected to the db ether_sql
[2018-09-10 21:51:13,390][INFO][ether_sql.session] Connected to Geth node
[2018-09-10 21:51:13,390][INFO][ether_sql.globals] Pushing the session PersonalGethSettings in local thread
Traceback (most recent call last):
  File "/home/ubuntu/ether_sql/envname/bin/ether_sql", line 11, in <module>
    load_entry_point('ether-sql', 'console_scripts', 'ether_sql')()
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/ether_sql/ether_sql/cli/__init__.py", line 91, in scrape_block_range
    mode=mode)
  File "/home/ubuntu/ether_sql/ether_sql/tasks/scrapper.py", line 66, in scrape_blocks
    add_block_number(block_number)
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/celery/local.py", line 193, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/ubuntu/ether_sql/envname/lib/python3.6/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/ether_sql/ether_sql/tasks/scrapper.py", line 143, in add_block_number
    timestamp=iso_timestamp)
  File "/home/ubuntu/ether_sql/ether_sql/models/receipts.py", line 60, in add_receipt
    status = bool(to_int(receipt_data['status']))
TypeError: 'NoneType' object is not subscriptable

same command should work with geth or is not yet enabled?