Skip to content

优化了一些常见的DataFrame操作方法;优化了过滤和匹配图数据库中节点的方法#148

Open
seer-xjc wants to merge 1 commit into
lynxworld:devfrom
seer-xjc:xjc_branch
Open

优化了一些常见的DataFrame操作方法;优化了过滤和匹配图数据库中节点的方法#148
seer-xjc wants to merge 1 commit into
lynxworld:devfrom
seer-xjc:xjc_branch

Conversation

@seer-xjc

@seer-xjc seer-xjc commented Dec 19, 2024

Copy link
Copy Markdown

小组名称:XY

链接:#148

成员1 xjc

分工 优化了 DefaultDataFrameOperator 类中的 DataFrameOperator 接口,对一些常见的 DataFrame 操作方法如选择(select)、分组(groupBy)、排序(orderBy)等做出改进。

改动文件 src\main\scala\org\grapheco\lynx\dataframe\DefaultDataFrameOperator.scala

具体优化内容:

  1. select 方法
    • 添加了 prepareColumns 辅助方法来缓存列索引和模式,避免每次调用 select 时都要重新计算。
  2. groupBy 方法
    • 使用 .view.flatMap 避免中间集合的创建,可以提高性能并减少内存使用。
    • 将聚合表达式的结果映射到最终输出,不在每个组上构建新的上下文,减少了上下文对象的数量。
  3. orderBy 方法
    • 在排序逻辑中,修改了比较逻辑,使得一旦找到两个元素不相等就可以立即返回比较结果,不用遍历整个 sortItem 序列。
  4. _ascCmp 方法
    • 修改为使用 find 方法,这样可以在找到第一个非零比较结果时立即停止迭代,提高了效率。

成员2 yy

分工 优化了通过多个条件来过滤和匹配图数据库中节点的方法

改动文件 src/main/scala/org/grapheco/lynx/runner/NodeFilter.scala

具体优化内容:

  1. 提取通用的 matchesProperty 方法,提高程序扩展性

将不同的属性操作符抽象为一个单独的 matchesProperty 函数。这样,当添加新操作符时,只需修改这个方法,而不需要修改 matches 方法中的每个 case 分支。

  1. 简化 matches 方法中的逻辑

提前检查 labels 是否匹配,避免不必要的属性匹配。如果没有 propOps,则默认所有属性与给定值相等,直接通过 node.property(propertyName).contains(value) 来匹配;如果有 propOps,调用 matchesProperty 方法来检查每个属性是否符合操作符的要求。

  1. 移除冗余的 Optionexists 调用

matchesProperty 中,使用 node.property(propertyName) 直接返回 SomeNone,避免多次调用 exists,使代码更简洁。

  1. 改进错误处理

对于不支持的操作符,抛出明确的异常,告诉开发者有一个不支持的操作符。这使得扩展时可以更加方便

@HC-teemo HC-teemo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice PR. Some of the codes don't seem to make much sense.
Also, if you don't mind, please test these changes (and if there is no relevant test code in the project please add it) because the Dataframe is quite core and needs to be strictly verified for correctness.

DataFrame(df.schema, () => sortedRecords.iterator)
}

private def _ascCmp(sortValue: Iterator[(LynxValue, LynxValue, Boolean)]): Boolean = {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function _ascCmp does not seem to be used.

val valueB = expressionEvaluator.eval(exp)(ctxB)
(valueA.compareTo(valueB), asc)
}
sortValues.exists { case (cmp, asc) => cmp != 0 && (cmp > 0) == asc }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem to make much sense here. Need to return asc judgement on the first different value. For example the following example: [cmp,asc]={(0,t),(1,f),(1,t)}. should return false at (1,f), but the exists function will return true, because (1,t).

val columnNameIndex: Map[String, Int] = df.columnsName.zipWithIndex.toMap
val newSchema: Seq[(String, LynxType)] = columns.map(column => column._2.getOrElse(column._1) -> sourceSchema(column._1))
val usedIndex: Seq[Int] = columns.map(_._1).map(columnNameIndex)
val (sourceSchema, _, usedIndices) = prepareColumns(df, columns)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be just extracting these few operations, and it's not that different.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants