Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
171 commits
Select commit Hold shift + click to select a range
5f17f21
Adding Dataset->CatalogueDCAT-AP and Observation and controlling some…
Nov 10, 2022
91ba51a
Correct save Dataset when there is no id, no content and generate the…
Nov 10, 2022
47c0821
Added some tests and corrected some code from tests
jicarretero Dec 9, 2022
dde437c
LICENSE
Jan 6, 2023
3c25ce5
system changed
Jan 6, 2023
83bbc7c
Adding Observation class with Error control over confStatus values
Jan 9, 2023
bca13e5
Adding ObsStatus management and testing class
Jan 10, 2023
b0ff0d0
Adding control of Decimal and UnitMult codes with unittests
Jan 12, 2023
15afb5d
Improve Code class redufing redundant code
Jan 12, 2023
497b092
Adding Frequency management and correction other issues
Jan 13, 2023
981f297
changes on datatypeconversion
jicarretero Jan 17, 2023
f8d9237
updated datatypeconversion to fit input data again
jicarretero Jan 17, 2023
21b5d4c
Modified structures generated to be compliant with ngsi-ld
jicarretero Jan 19, 2023
3952189
Resolve concept.py error with type relationship and add more function…
Jan 23, 2023
5bc4692
Normalising the rest of pending data in catalogue.py
Jan 23, 2023
e1cf26f
Translate __generate_property__ from catalogue to commonclass
Jan 25, 2023
a0cecf9
Update file permissions
Jan 25, 2023
8eb3e2b
Adding the list of dimensions into the Observation
Jan 27, 2023
e6a6b33
Homogeneize all properties as normalized JSON-LD representation
Feb 7, 2023
f891232
Bump lark from 1.1.3 to 1.1.5
dependabot[bot] Feb 10, 2023
a8b6ae9
Bump uvicorn from 0.19.0 to 0.21.0
dependabot[bot] Mar 10, 2023
0fc8bbb
Corrected boolean tests and requirements.txt
jicarretero Mar 13, 2023
2e8bbfd
Bump fastapi from 0.85.1 to 0.94.1
dependabot[bot] Mar 14, 2023
119c074
Merge pull request #72 from flopezag/dependabot/pip/fastapi-0.94.1
flopezag Mar 15, 2023
5c06cf6
Merge branch 'master' into dependabot/pip/uvicorn-0.21.0
flopezag Mar 15, 2023
e3fb569
Bump python-multipart from 0.0.5 to 0.0.6
dependabot[bot] Mar 15, 2023
0fd87f4
Merge pull request #70 from flopezag/dependabot/pip/uvicorn-0.21.0
flopezag Mar 15, 2023
16e3fee
Merge branch 'master' into dependabot/pip/python-multipart-0.0.6
flopezag Mar 15, 2023
fb23d3d
Merge pull request #68 from flopezag/dependabot/pip/python-multipart-…
flopezag Mar 15, 2023
2cdb7c8
Merge pull request #63 from flopezag/dependabot/pip/lark-1.1.5
flopezag Mar 15, 2023
f4872f5
Bump requests from 2.28.1 to 2.28.2
dependabot[bot] Mar 15, 2023
3201561
Merge pull request #60 from flopezag/dependabot/pip/requests-2.28.2
flopezag Mar 15, 2023
6ddf54f
Bump lark from 1.1.3 to 1.1.5
dependabot[bot] Mar 15, 2023
fc15552
Bump requests from 2.28.1 to 2.28.2
dependabot[bot] Mar 15, 2023
018b320
Bump uvicorn from 0.19.0 to 0.21.1
dependabot[bot] Mar 16, 2023
fbacd12
Bump fastapi from 0.85.1 to 0.95.0
dependabot[bot] Mar 20, 2023
d36515e
Merge pull request #79 from flopezag/dependabot/pip/fastapi-0.95.0
flopezag Mar 21, 2023
827a6dc
Merge branch 'master' into dependabot/pip/uvicorn-0.21.1
flopezag Mar 21, 2023
c78144d
Bump python-multipart from 0.0.5 to 0.0.6
dependabot[bot] Mar 21, 2023
b96c4b9
Merge pull request #75 from flopezag/dependabot/pip/requests-2.28.2
flopezag Mar 21, 2023
93395b5
Merge pull request #78 from flopezag/dependabot/pip/uvicorn-0.21.1
flopezag Mar 21, 2023
d15a9a8
Merge pull request #77 from flopezag/dependabot/pip/python-multipart-…
flopezag Mar 21, 2023
47a9375
Merge pull request #73 from flopezag/dependabot/pip/lark-1.1.5
flopezag Mar 21, 2023
bccccce
Bump fastapi from 0.85.1 to 0.95.0
dependabot[bot] Mar 21, 2023
7049c0e
Bump uvicorn from 0.19.0 to 0.21.1
dependabot[bot] Mar 21, 2023
3207539
Update rdflib requirement from ~6.2.0 to >=6.2,<6.4
dependabot[bot] Mar 21, 2023
a571975
Merge pull request #81 from flopezag/dependabot/pip/uvicorn-0.21.1
flopezag Mar 22, 2023
b57090e
Merge branch 'master' into dependabot/pip/fastapi-0.95.0
flopezag Mar 22, 2023
a8319d6
Merge pull request #82 from flopezag/dependabot/pip/rdflib-gte-6.2-an…
flopezag Mar 22, 2023
0276b8d
Merge pull request #80 from flopezag/dependabot/pip/fastapi-0.95.0
flopezag Mar 22, 2023
ecb0c51
Added some tests for common and slightly changed commonclass.py
jicarretero Mar 22, 2023
04bc59b
Update rdflib requirement from ~=6.2.0 to >=6.2,<6.4
dependabot[bot] Mar 22, 2023
ea83040
Added another test for lists
jicarretero Apr 5, 2023
5dd1379
solved CB writings
jicarretero Apr 6, 2023
f3f4861
Bump loguru from 0.6.0 to 0.7.0
dependabot[bot] Apr 10, 2023
9f7f883
Bump fastapi from 0.95.0 to 0.95.1
dependabot[bot] Apr 13, 2023
9f69c58
Merge pull request #85 from flopezag/dependabot/pip/fastapi-0.95.1
flopezag Apr 21, 2023
8424dda
Merge pull request #84 from flopezag/dependabot/pip/loguru-0.7.0
flopezag Apr 21, 2023
cc05c72
Merge pull request #83 from flopezag/dependabot/pip/rdflib-gte-6.2-an…
flopezag Apr 21, 2023
c692e36
Bump fastapi from 0.95.0 to 0.95.1
dependabot[bot] Apr 21, 2023
0b0ee58
Bump loguru from 0.6.0 to 0.7.0
dependabot[bot] Apr 21, 2023
802101e
Merge pull request #87 from flopezag/dependabot/pip/loguru-0.7.0
flopezag Apr 24, 2023
d729a4a
Merge pull request #86 from flopezag/dependabot/pip/fastapi-0.95.1
flopezag Apr 24, 2023
b446d01
Resolution of several bugs
Apr 24, 2023
cacd7a1
Resolution some bugs
Apr 25, 2023
9fa1b66
Bump requests from 2.28.2 to 2.29.0
dependabot[bot] Apr 26, 2023
fd7963d
Bump uvicorn from 0.21.1 to 0.22.0
dependabot[bot] Apr 28, 2023
f445d39
Merge pull request #96 from flopezag/dependabot/pip/uvicorn-0.22.0
flopezag May 4, 2023
b06527d
Merge pull request #95 from flopezag/dependabot/pip/requests-2.29.0
flopezag May 4, 2023
4a466cb
Resolving testing-feature conflicts
jicarretero May 4, 2023
952f6a4
Merge branch 'feature/observation' into feature/testing
jicarretero May 4, 2023
4ecce40
Merge pull request #97 from flopezag/feature/testing
flopezag May 4, 2023
a1eb96f
Merge branch 'feature/observation' into ngsild
jicarretero May 4, 2023
a383392
Merge pull request #98 from flopezag/ngsild
flopezag May 4, 2023
5d796b1
Bump uvicorn from 0.21.1 to 0.22.0
dependabot[bot] May 4, 2023
5222d35
Bump requests from 2.28.2 to 2.30.0
dependabot[bot] May 4, 2023
60a7460
Repaired broken tests after last merge
jicarretero May 16, 2023
e359589
Merge pull request #101 from flopezag/debt/repair-tests
flopezag May 16, 2023
d16bb23
Merge pull request #100 from flopezag/dependabot/pip/requests-2.30.0
flopezag May 16, 2023
9bd3a91
Merge pull request #99 from flopezag/dependabot/pip/uvicorn-0.22.0
flopezag May 16, 2023
f681df0
Bump requests from 2.28.2 to 2.31.0
dependabot[bot] May 22, 2023
b3f25fd
Bump fastapi from 0.95.1 to 0.97.0
dependabot[bot] Jun 12, 2023
a5b6ddf
Merge pull request #106 from flopezag/dependabot/pip/fastapi-0.97.0
flopezag Jun 14, 2023
4e6eb55
Merge pull request #104 from flopezag/dependabot/pip/requests-2.31.0
flopezag Jun 14, 2023
6d840f8
Bump requests from 2.28.2 to 2.31.0
dependabot[bot] Jun 14, 2023
30548cc
Merge pull request #107 from flopezag/dependabot/pip/requests-2.31.0
flopezag Jun 14, 2023
6ece50f
Update requirements file
Jun 14, 2023
26434c0
Bump fastapi from 0.95.1 to 0.97.0
dependabot[bot] Jun 14, 2023
54580fb
Merge pull request #108 from flopezag/dependabot/pip/fastapi-0.97.0
flopezag Jun 15, 2023
622ed42
Merge branch 'feature/observation' of github.com:flopezag/IoTAgent-Tu…
Jun 15, 2023
a5752f2
Remove VSCode configuration file
Jun 15, 2023
fa366dc
Resolving sdmx-dimensions issues
Jun 15, 2023
7eb4828
Implement multi observations support
Jun 16, 2023
c56e4c4
Adding sdmx-dimensions properties
Jun 16, 2023
797305e
Add SDMX-Concepts, SDMX-Dimensions, SDMX-ConceptSchemas, and DCAT-AP:…
Jun 16, 2023
d8ff532
Adding the sdmx-attributes obsStatus and confStatus
Jun 19, 2023
780b674
Refactor exception classes to the proper python package
Jun 19, 2023
55a46d5
Refactoring smdx-attributes and sdmx-dimensions
Jun 19, 2023
db01933
Refactoring finished for sdmx-attributes and sdmx-dimensions
Jun 19, 2023
8276b2a
Merge pull request #121 from flopezag/feature/observation
jicarretero Jun 19, 2023
9b7ecfe
Update sdmx-attributes and sdmx-concepts
Jun 20, 2023
2820f16
Updated data generation and integrated CB
jicarretero Jun 20, 2023
ded7aba
Integrated cb-connector
jicarretero Jun 20, 2023
8f04f8c
Correct sdmx-measure, it is not a unitStatMeasure
Jun 20, 2023
fbf113f
Added docker file
jicarretero Jun 20, 2023
2544f67
Added readme.md and solved some bugs
jicarretero Jun 20, 2023
f7ee268
Changed a bit Readme.md and set a more convenient branch to pull when…
jicarretero Jun 20, 2023
51c8d03
Resolve the prefix resolution in the DimensionProperty class
Jun 20, 2023
6a2eb47
Resolution of prefix and context in the Concept class
Jun 20, 2023
2c669f1
Resolve prefixes in ConceptSchema, SDMX-Concepts, SDMX-Attributes, an…
Jun 20, 2023
ad0dcfe
Resolve the Catalogue prefix management and update class following th…
Jun 20, 2023
cd39ee1
Resolution of the Distribution class (prefix) and fixing data model r…
Jun 21, 2023
7256fce
changed dockerfile
jicarretero Jun 21, 2023
2dc150f
changed version for other things
jicarretero Jun 21, 2023
697e105
change documentation
jicarretero Jun 21, 2023
9b5c1e4
finished all changes in the review
jicarretero Jun 21, 2023
093dc93
Resolution Dataset prefix and quality code checking
Jun 21, 2023
63e157d
Adding measureType as a new dimension defined externally to the parse…
Jun 21, 2023
b5afa44
Correction a comment
Jun 21, 2023
6e84d87
Changed some comments in Readme and the dockerfile itself
jicarretero Jun 21, 2023
4abd42a
changed image for imagine
jicarretero Jun 21, 2023
43e1e33
changed magic numbers for fastapip.HTTP_XXX values
jicarretero Jun 21, 2023
a8e8dbe
Merge pull request #124 from flopezag/integration-cb-httpserver
flopezag Jun 21, 2023
2e100ae
Merge branch 'develop' into feature/observation
flopezag Jun 21, 2023
b52992f
Merge pull request #130 from flopezag/feature/observation
jicarretero Jun 22, 2023
dd08705
changed broken test
jicarretero Jun 22, 2023
a927bff
Some property name changed in file ngsild_connector.py and was proper…
jicarretero Jun 22, 2023
6db565e
Resolution of a bug in Distribution class
Jun 22, 2023
782f268
Merge branch 'feature/observation' of github.com:flopezag/IoTAgent-Tu…
Jun 22, 2023
c98e87e
Merge pull request #131 from flopezag/feature/observation
jicarretero Jun 22, 2023
e32bd49
Added logs to ngsild in case of HTTP/400 to visualize the problematic…
jicarretero Jun 22, 2023
df93527
Merge branch 'develop' into test-development-merge
jicarretero Jun 22, 2023
2741531
Changed api.yaml file describing the operations and results returned …
jicarretero Jun 22, 2023
45b1414
Removed on line. It was incorrect
jicarretero Jun 22, 2023
1b1f6d0
Merge pull request #133 from flopezag/test-development-merge
flopezag Jun 22, 2023
3cdafa3
Review bug #132
Jun 22, 2023
54bbdd4
Merge pull request #135 from flopezag/feature/observation
jicarretero Jun 22, 2023
52ce266
Bump fastapi from 0.97.0 to 0.98.0
dependabot[bot] Jun 22, 2023
aa832e9
Review testing process
Jun 23, 2023
7b0ad83
Merge pull request #136 from flopezag/dependabot/pip/fastapi-0.98.0
flopezag Jun 23, 2023
f47dcbc
Bump fastapi from 0.97.0 to 0.98.0
dependabot[bot] Jun 23, 2023
1345e1e
Merge pull request #137 from flopezag/dependabot/pip/fastapi-0.98.0
flopezag Jun 24, 2023
867956b
Complete test_examples execution tests
Jun 28, 2023
8cc386a
Bump fastapi from 0.98.0 to 0.99.0
dependabot[bot] Jun 30, 2023
e720116
Merge pull request #138 from flopezag/dependabot/pip/fastapi-0.99.0
flopezag Jul 3, 2023
eb8f5c2
Bump fastapi from 0.98.0 to 0.100.0
dependabot[bot] Jul 7, 2023
3914f72
Merge pull request #140 from flopezag/dependabot/pip/fastapi-0.100.0
flopezag Jul 8, 2023
0466480
Bump fastapi from 0.98.0 to 0.100.0
dependabot[bot] Jul 10, 2023
aa10d18
Merge pull request #141 from flopezag/dependabot/pip/fastapi-0.100.0
flopezag Jul 11, 2023
33bd735
Bump lark from 1.1.5 to 1.1.6
dependabot[bot] Jul 17, 2023
64bc2de
Bump uvicorn from 0.22.0 to 0.23.0
dependabot[bot] Jul 17, 2023
2fe32c1
Merge pull request #143 from flopezag/dependabot/pip/uvicorn-0.23.0
flopezag Jul 18, 2023
b1e1769
Merge pull request #142 from flopezag/dependabot/pip/lark-1.1.6
flopezag Jul 18, 2023
958ab9f
Bump lark from 1.1.5 to 1.1.6
dependabot[bot] Jul 18, 2023
a9ac151
Bump uvicorn from 0.22.0 to 0.23.1
dependabot[bot] Jul 18, 2023
f28d258
Merge pull request #145 from flopezag/dependabot/pip/uvicorn-0.23.1
flopezag Jul 19, 2023
b3b5f17
Merge pull request #144 from flopezag/dependabot/pip/lark-1.1.6
flopezag Jul 19, 2023
97fabd0
Bump uvicorn from 0.22.0 to 0.23.1
dependabot[bot] Jul 19, 2023
accd4d8
Merge pull request #146 from flopezag/dependabot/pip/uvicorn-0.23.1
flopezag Jul 20, 2023
e04a2d3
Bump lark from 1.1.6 to 1.1.7
dependabot[bot] Jul 20, 2023
ec01b04
Merge pull request #147 from flopezag/dependabot/pip/lark-1.1.7
flopezag Jul 21, 2023
af5904f
Bump lark from 1.1.6 to 1.1.7
dependabot[bot] Jul 21, 2023
c41e106
Merge pull request #148 from flopezag/dependabot/pip/lark-1.1.7
flopezag Jul 24, 2023
c40ae0f
Update requirements.txt and resolve an issue in Concept Schema when d…
Jul 24, 2023
7729064
Linter and type checking
Jul 25, 2023
ac90598
Update files with line-length to 120 (linter)
Jul 25, 2023
05ba69f
Final configuration of the poetry and tox
Jul 25, 2023
816fea8
Update gitignore to add the environment configuration file for docker
Jul 25, 2023
6f0c760
Resolveing merge issues
Jul 25, 2023
f8ba76b
Prepare new release
Jul 25, 2023
3f497b8
Resolving bug in Property with special Relationships
Oct 9, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# .coveragerc to control coverage.py
[run]
branch = True
omit =
./docs/*
./docker/*
./examples/*
./grammar/*
./logs/*
./tests/*
./dist/*
./images/*
./output/*

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
# Have to re-enable the standard pragma
pragma: no cover

# Don't complain about missing debug-only code:
def __repr__
if self\.debug

# Don't complain if tests don't hit defensive assertion code:
raise AssertionError
raise NotImplementedError

# Don't complain if non-runnable code isn't run:
if 0:
if __name__ == .__main__.:

# Don't complain about abstract methods, they aren't run:
@(abc\.)?abstractmethod

ignore_errors = True

[html]
directory = coverage_html_report
3 changes: 1 addition & 2 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
.venv/
doc/
docs/
examples/
test/
*.jsonld
Expand All @@ -8,7 +8,6 @@ tox.ini
test-requirements.txt
README.md
.stestr.conf
.pre-commit-config.yaml
.gitignore
logs/access.log
.git/
Expand Down
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,6 @@ celerybeat.pid
*.sage.py

# Environments
.env
.venv
env/
venv/
Expand Down Expand Up @@ -130,8 +129,13 @@ dmypy.json

# IDE
.idea
.vscode

# Temporal generated JSON-LD documents
*.jsonld
output
output/*.jsonld
tests/test1.py

# logs folder
logs
19 changes: 9 additions & 10 deletions agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@
from sdmx2jsonld.transform.parser import Parser
from api.server import launch
from sdmx2jsonld.exceptions import UnexpectedEOF, UnexpectedInput, UnexpectedToken
from ngsild.ngsild_connector import NGSILDConnector

if __name__ == '__main__':
if __name__ == "__main__":
args = parse_cli()

if args['run'] is True:
file_in = args['--input']
generate_files = args['--output']
if args["run"] is True:
file_in = args["--input"]
generate_files = args["--output"]

my_parser = Parser()

Expand All @@ -43,10 +44,8 @@
except UnexpectedEOF as e:
print(e)

elif args['server'] is True:
port = int(args['--port'])
host = args['--host']
elif args["server"] is True:
port = int(args["--port"])
host = args["--host"]

launch(app="api.server:application",
host=host,
port=port)
launch(app="api.server:application", host=host, port=port)
55 changes: 21 additions & 34 deletions api/custom_logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@

class InterceptHandler(Handler):
loglevel_mapping = {
50: 'CRITICAL',
40: 'ERROR',
30: 'WARNING',
20: 'INFO',
10: 'DEBUG',
0: 'NOTSET',
50: "CRITICAL",
40: "ERROR",
30: "WARNING",
20: "INFO",
10: "DEBUG",
0: "NOTSET",
}

def emit(self, record):
Expand All @@ -49,46 +49,32 @@ def emit(self, record):
frame = frame.f_back
depth += 1

log = logger.bind(request_id='app')
log.opt(
depth=depth,
exception=record.exc_info
).log(level,record.getMessage())
log = logger.bind(request_id="app")
log.opt(depth=depth, exception=record.exc_info).log(level, record.getMessage())


class CustomizeLogger:

@classmethod
def make_logger(cls, config_path: Path):

config = cls.load_logging_config(config_path)
logging_config = config.get('logger')
logging_config = config.get("logger")

logger = cls.customize_logging(
filepath=logging_config.get('path'),
level=logging_config.get('level'),
retention=logging_config.get('retention'),
rotation=logging_config.get('rotation'),
format=logging_config.get('format'))
filepath=logging_config.get("path"),
level=logging_config.get("level"),
retention=logging_config.get("retention"),
rotation=logging_config.get("rotation"),
format=logging_config.get("format"),
)

return logger

@classmethod
def customize_logging(cls,
filepath: Path,
level: str,
rotation: str,
retention: str,
format: str):
def customize_logging(cls, filepath: Path, level: str, rotation: str, retention: str, format: str):

logger.remove()

logger.add(
sys.stdout,
enqueue=True,
backtrace=True,
level=level.upper(),
format=format)
logger.add(sys.stdout, enqueue=True, backtrace=True, level=level.upper(), format=format)

logger.add(
str(filepath),
Expand All @@ -97,12 +83,13 @@ def customize_logging(cls,
enqueue=True,
backtrace=True,
level=level.upper(),
format=format)
format=format,
)

basicConfig(handlers=[InterceptHandler()], level=0)
getLogger("uvicorn.access").handlers = [InterceptHandler()]

for _log in ['uvicorn', 'uvicorn.error', 'uvicorn.access', 'fastapi']:
for _log in ["uvicorn", "uvicorn.error", "uvicorn.access", "fastapi"]:
_logger = getLogger(_log)
_logger.handlers = [InterceptHandler()]

Expand All @@ -113,4 +100,4 @@ def load_logging_config(cls, config_path):
config = None
with open(config_path) as config_file:
config = load(config_file)
return config
return config
Loading