优化了一些常见的DataFrame操作方法;优化了过滤和匹配图数据库中节点的方法#148
Conversation
HC-teemo
left a comment
There was a problem hiding this comment.
Very nice PR. Some of the codes don't seem to make much sense.
Also, if you don't mind, please test these changes (and if there is no relevant test code in the project please add it) because the Dataframe is quite core and needs to be strictly verified for correctness.
| DataFrame(df.schema, () => sortedRecords.iterator) | ||
| } | ||
|
|
||
| private def _ascCmp(sortValue: Iterator[(LynxValue, LynxValue, Boolean)]): Boolean = { |
There was a problem hiding this comment.
The function _ascCmp does not seem to be used.
| val valueB = expressionEvaluator.eval(exp)(ctxB) | ||
| (valueA.compareTo(valueB), asc) | ||
| } | ||
| sortValues.exists { case (cmp, asc) => cmp != 0 && (cmp > 0) == asc } |
There was a problem hiding this comment.
It doesn't seem to make much sense here. Need to return asc judgement on the first different value. For example the following example: [cmp,asc]={(0,t),(1,f),(1,t)}. should return false at (1,f), but the exists function will return true, because (1,t).
| val columnNameIndex: Map[String, Int] = df.columnsName.zipWithIndex.toMap | ||
| val newSchema: Seq[(String, LynxType)] = columns.map(column => column._2.getOrElse(column._1) -> sourceSchema(column._1)) | ||
| val usedIndex: Seq[Int] = columns.map(_._1).map(columnNameIndex) | ||
| val (sourceSchema, _, usedIndices) = prepareColumns(df, columns) |
There was a problem hiding this comment.
This seems to be just extracting these few operations, and it's not that different.
小组名称:XY
链接:#148
成员1 xjc
分工 优化了
DefaultDataFrameOperator类中的DataFrameOperator接口,对一些常见的DataFrame操作方法如选择(select)、分组(groupBy)、排序(orderBy)等做出改进。改动文件 src\main\scala\org\grapheco\lynx\dataframe\DefaultDataFrameOperator.scala
具体优化内容:
prepareColumns辅助方法来缓存列索引和模式,避免每次调用select时都要重新计算。.view.flatMap避免中间集合的创建,可以提高性能并减少内存使用。sortItem序列。find方法,这样可以在找到第一个非零比较结果时立即停止迭代,提高了效率。成员2 yy
分工 优化了通过多个条件来过滤和匹配图数据库中节点的方法
改动文件 src/main/scala/org/grapheco/lynx/runner/NodeFilter.scala
具体优化内容:
matchesProperty方法,提高程序扩展性将不同的属性操作符抽象为一个单独的
matchesProperty函数。这样,当添加新操作符时,只需修改这个方法,而不需要修改matches方法中的每个case分支。matches方法中的逻辑提前检查
labels是否匹配,避免不必要的属性匹配。如果没有propOps,则默认所有属性与给定值相等,直接通过node.property(propertyName).contains(value)来匹配;如果有propOps,调用matchesProperty方法来检查每个属性是否符合操作符的要求。Option和exists调用在
matchesProperty中,使用node.property(propertyName)直接返回Some或None,避免多次调用exists,使代码更简洁。对于不支持的操作符,抛出明确的异常,告诉开发者有一个不支持的操作符。这使得扩展时可以更加方便