How to setting proxy authorization with username/passwd in ubuntu-server 18.04 env

[luorixiangyang]

How to setting proxy authorization with username/passwd in ubuntu-server 18.04 env?
I found lots of example but dont reslove my requirement to scrape the web like : (https://developer.apple.com/documentation/accelerate/bnns/shape/3656199-init)

thanks!

[rbri]

Looks like there is something missing ;-) will have a deeper look

[rbri]

As a workaround can you please try something like

String PROXY_HOST = ....;
int PROXY_PORT = .....

WebDriver webDriver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) {
    @Override
    protected WebClient modifyWebClient(WebClient client) {
        final WebClient webClient = super.modifyWebClient(client);

        webClient.getOptions().setProxyConfig(new ProxyConfig(PROXY_HOST, PROXY_PORT, null));
        final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider();
        credentialsProvider.addCredentials("username", "password", PROXY_HOST, PROXY_PORT);

       return webClient;
    }
};

[luorixiangyang]

As a workaround can you please try something like

String PROXY_HOST = ....;
int PROXY_PORT = .....

WebDriver webDriver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) {
    @Override
    protected WebClient modifyWebClient(WebClient client) {
        final WebClient webClient = super.modifyWebClient(client);

        webClient.getOptions().setProxyConfig(new ProxyConfig(PROXY_HOST, PROXY_PORT, null));
        final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider();
        credentialsProvider.addCredentials("username", "password", PROXY_HOST, PROXY_PORT);

       return webClient;
    }
};

Here is the detail infos:
pom.xml dependency like below:
...

org.seleniumhq.selenium
selenium-java
4.10.0


org.seleniumhq.selenium
htmlunit-driver
4.10.0

...

source code like below:
public static WebDriver createProxyWebDriver() {
String PROXY_HOST = ProxyHost;
int PROXY_PORT = ProxyPort;

    // config webDriver with proxies
    WebDriver webDriver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) {
        @Override
        protected WebClient modifyWebClient(WebClient client) {
            final WebClient webClient = super.modifyWebClient(client);

            webClient.getOptions().setProxyConfig(new ProxyConfig(PROXY_HOST, PROXY_PORT, null));
            final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient
                    .getCredentialsProvider();
            credentialsProvider.addCredentials(ProxyUser, ProxyPass, PROXY_HOST, PROXY_PORT, null);

            return webClient;
        }
    };
    return webDriver;
}

public static String getPageOnDynamicWeb(String url) {
WebDriver client = createProxyWebDriver();
client.get(url);
String response = client.getPageSource();
client.close();
return response;
}

public static void main(String[] args) throws Exception {
String response = "";
String url = "https://developer.apple.com/documentation/accelerate/bnns/shape/3656199-init"; // the target url
response = getPageOnDynamicWeb(url);
ClearInnerToWriteFile(
"/home/luori/_fly/workspaces/javaworkspace/selenium-base/logs/apple_api_page_html.html",
response);
}

Run before code will take exception like below:
......
Caused by: net.sourceforge.htmlunit.corejs.javascript.EvaluatorException: invalid property id (https://developer.apple.com/tutorials/js/chunk-vendors.fc64ed7e.js#10)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory$HtmlUnitErrorReporter.error(HtmlUnitContextFactory.java:435)
at net.sourceforge.htmlunit.corejs.javascript.Parser.addError(Parser.java:257)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:336)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:327)
at net.sourceforge.htmlunit.corejs.javascript.Parser.reportError(Parser.java:320)
at net.sourceforge.htmlunit.corejs.javascript.Parser.objectLiteral(Parser.java:3499)
......

[luorixiangyang]

Environment:
Ubuntu-server 18.04
google-chrome： Google Chrome 114.0.5735.133
ChromeDriver：114.0.5735.90
JDK：1.8.0_271

[luorixiangyang]

Please try target url :https://developer.apple.com/documentation/accelerate/bnns/shape/3656199-init to test the correct approach .
Thanks!

[luorixiangyang]

I need to point out ：(https://developer.apple.com/documentation/accelerate/bnns/shape/3656199-init) is dynamic web content， need excute javascript file on scrape process. I can get the static web content but can't catch the dynamic parts.

[rbri]

Had a deeper look and there are several problems with this page. Long story short - HtmlUnit does not support the whole modern javascript syntay (because it is based on Rhino). We are working on improving this but i fear there is no real progress until the end of this year.

Two options: help us to improve Rhino or use selenium with real browsers

[luorixiangyang]

Had a deeper look and there are several problems with this page. Long story short - HtmlUnit does not support the whole modern javascript syntay (because it is based on Rhino). We are working on improving this but i fear there is no real progress until the end of this year.

Two options: help us to improve Rhino or use selenium with real browsers

Got it! I also check if i can make contribution on HtmlUitl to improve this issue.