FileUpload OutOfDirectMemoryError

[markbanyang]

当我在上传大文件的时候会发生OOM
下面是文件上传代码:
`

@RequestMapping(value = "/stream-upload-test")

  public ResponseEntity<String> upload(HttpServletRequest request, HttpServletResponse response)
        throws IOException, FileUploadException {
    boolean isMultipart = ServletFileUpload.isMultipartContent(request);
    if (isMultipart) {
        ServletFileUpload upload = new ServletFileUpload();

        Map<String, String> params = new HashMap<>();
        InputStream is = null;
        FileItemIterator iter = upload.getItemIterator(request);
        while (iter.hasNext()) {
            FileItemStream item = iter.next();
            if (!item.isFormField()) {
                is = item.openStream();
                try {
                    int i = 0;
                    byte bb [] = new byte[4096];
                    while((i = is.read(bb)) != -1){
                        System.out.println(Arrays.toString(bb));
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                } finally {
                    if (null != is) {
                        is.close();
                    }
                }
            } else {

                String fieldName = item.getFieldName();
                String value = Streams.asString(item.openStream());
                System.out.println(fieldName + " : " + value);
            }
        }
    }
    return new ResponseEntity<String>(HttpStatus.OK);
}

`

错误信息截图如下：

[wangzihaogithub]

我看下 , 估计明天上午解决

[wangzihaogithub]

解决了. 在这次提交, d65fe24
已经合并到master分支了, 也已经提交到maven中央仓库了. 过两天就同步到中央仓库了.

原因是: 错误的将netty的HttpPostMultipartRequestDecoder 的 discardThreshold参数设置成了Integer.MAX_VALUE导致的.

public int getDiscardThreshold(){
int discardThreshold = 0;
if(multipartConfigElement != null) {
discardThreshold = (int)multipartConfigElement.getMaxFileSize();
}
if(discardThreshold <= 0){
discardThreshold = Integer.MAX_VALUE; //就是这里, 这个参数是控制复制文件的缓冲区, 我现在改成了不设置, 即netty默认的最大10M.
}
return discardThreshold;
}

[markbanyang]

我验证了下，上传文件OOM的问题已经解决了。

[markbanyang]

另外我在测试下载大文件的时候，耗时比tomcat长很多，从抓取的数据包发现，下载文件的时候发送一部分包后，会等待一段时间才会继续发送下一个包：

下面是我的测试代码：
`@RequestMapping("/file-download-test")

public ResponseEntity<String> downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception {
    String fileName = "CentOS-7-x86_64-DVD-2003.iso";
    String filePath = "./" + fileName;
    handleDownloadStream(fileName, filePath, request, response);
    return new ResponseEntity<>(HttpStatus.OK);
}

public void handleDownloadStream(String fileName, String filePath, HttpServletRequest request, HttpServletResponse res) throws IOException {
byte[] buffer = new byte[4 * 1024];
OutputStream os = null;
FileInputStream fStream = null;
try {
os = new BufferedOutputStream(res.getOutputStream());
res.reset();
String agent = request.getHeader("User-Agent");
if (agent == null) {
return;
}
agent = agent.toUpperCase();

        //ie浏览器,火狐,Edge浏览器
        if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
            fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
        } else {
            fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
        }
        //safari RFC 5987标准
        if (agent.indexOf("SAFARI") > -1) {
            res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
        } else {
            res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
        }
        File file = new File(filePath);
        res.setContentType("application/octet-stream");
        res.setCharacterEncoding("UTF-8");
        res.setContentLength((int) file.length());
        fStream = new FileInputStream(file);
        int length = 0;
        while ((length = fStream.read(buffer)) != -1) {
            os.write(buffer, 0, length);
        }
        os.flush();

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(fStream);
    }
}`

[wangzihaogithub]

这是由这个参数控制的， 你可以改成1KB。默认达到是1M后，或者主动调用flush方法，再给客户端发数据。

server.netty.http-servlet.response-max-buffer-size=1024

[wangzihaogithub]

我下个版本把默认值改小点，另外， 你可以这样发文件，性能会好一些，因为用的是netty的零拷贝

NettyOutputStream nettyOutputStream = ((NettyOutputStream)response.getOutputStream())；
nettyOutputStream.write(new File("/home/temp.json"));

[markbanyang]

好的 我尝试下

[markbanyang]

文件下载是否存在内存不释放的问题，配置启动内存-Xmx512M -XX:MaxDirectMemorySize=512M，下载一个不到2G文件会卡住，并且会将分配的内存耗尽。

[wangzihaogithub]

可以看下代码吗？ 我用的你第一次的代码复现不出来。

master分支的这个类 com.github.netty.http.example.HttpController， 是我用的例子， 不知道一样不

`

@RequestMapping("/downloadFile")
public ResponseEntity downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception {
String fileName = "CentOS-7-x86_64-DVD-2003.iso";

    byte[] file = new byte[1024 * 1024 * 7];
    for (int i = 0; i < file.length; i++) {
        file[i] = (byte) i;
    }
    handleDownloadStream(fileName, new ByteArrayInputStream(file), request, response);
    return new ResponseEntity<>(HttpStatus.OK);
}

public void handleDownloadStream(String fileName, InputStream inputStream, HttpServletRequest request, HttpServletResponse res) throws IOException {
    byte[] buffer = new byte[4 * 1024];
    OutputStream os = null;
    try {
        os = new BufferedOutputStream(res.getOutputStream());
        res.reset();
        String agent = request.getHeader("User-Agent");
        if (agent == null) {
            return;
        }
        agent = agent.toUpperCase();

        //ie浏览器,火狐,Edge浏览器
        if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
            fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
        } else {
            fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
        }
        //safari RFC 5987标准
        if (agent.contains("SAFARI")) {
            res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
        } else {
            res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
        }
        res.setContentType("application/octet-stream");
        res.setCharacterEncoding("UTF-8");
        res.setContentLength(inputStream.available());
        int length = 0;
        while ((length = inputStream.read(buffer)) != -1) {
            os.write(buffer, 0, length);
        }
        os.flush();

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(inputStream);
    }
}

`

[wangzihaogithub]

辛苦发一下吧 感谢

[markbanyang]

好的，稍等下。

[markbanyang]

`@RequestMapping("/file-download-test")
public ResponseEntity downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception {
String fileName = "CentOS-7-x86_64-DVD-2003.iso";
String filePath = "./" + fileName;
handleDownloadStream(fileName, filePath, request, response);
return new ResponseEntity<>(HttpStatus.OK);
}

public void handleDownloadStream(String fileName, String filePath, HttpServletRequest request, HttpServletResponse res) throws IOException {
    byte[] buffer = new byte[4 * 1024];
    OutputStream os = null;
    FileInputStream fStream = null;
    try {
        os = new BufferedOutputStream(res.getOutputStream());
        res.reset();
        String agent = request.getHeader("User-Agent");
        if (agent == null) {
            return;
        }
        agent = agent.toUpperCase();

        //ie浏览器,火狐,Edge浏览器
        if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
            fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
        } else {
            fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
        }
        //safari RFC 5987标准
        if (agent.indexOf("SAFARI") > -1) {
            res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
        } else {
            res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
        }
        File file = new File(filePath);
        res.setContentType("application/octet-stream");
        res.setCharacterEncoding("UTF-8");
        fStream = new FileInputStream(file);
        int length = 0;
        while ((length = fStream.read(buffer)) != -1) {
            os.write(buffer, 0, length);
            os.flush();
        }
        os.flush();

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(fStream);
    }
}`

被下载的文件大小大概4个G
jvm启动参数：

[wangzihaogithub]

发的文件好像坏了， 看不到

[wangzihaogithub]

要不贴代码吧， 就需要启动参数和代码片段就行

[markbanyang]

我发你邮箱了