Skip to main content

JAVA: How can I download an HTML file from a site that requires cookies enabled?

Image
By User



I'm trying to download an HTML file from a site. I'm using the following simple method:







URL url = new URL("here goes the link to the html file");

 BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));

 String htmlfile = "";

 String temp;

 while ((temp = br.readLine()) != null) {

       htmlfile+= temp;

 }







The problem is that I get the following String in the htmlfile variable:







The installation of ... requires the acceptance of a cookie by your browser    

software.    The cookie is used to ensure that you and only you are 

able to access information ....







In other words, I need to somewhat enable cookies when opening a stream from the url. Is it possible to achieve this by using URL or do I need a different method? Thanks in advance


Labels:

Comments

  1. Tips For AllFebruary 28, 2012 at 7:19 AM

    If you use a good library like Apache HttpComponents

    http://hc.apache.org/index.html

    it takes care of cookie-management for you.

    ReplyDelete
  2. Tips For AllFebruary 28, 2012 at 7:19 AM

    You can use addRequestProperty() to set a cookie on a URLConnection object, e.g.

    URL url = new URL("here goes the link to the html file");
    URLConnection connection = url.openConnection();
    connection.addRequestProperty("Cookie", "here goes the cookie");
    BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));

    ReplyDelete

Post a Comment

Popular posts from this blog

Why is this Javascript much *slower* than its jQuery equivalent?

By User
I have a HTML list of about 500 items and a "filter" box above it. I started by using jQuery to filter the list when I typed a letter (timing code added later): $('#filter').keyup( function() { var jqStart = (new Date).getTime(); var search = $(this).val().toLowerCase(); var $list = $('ul.ablist > li'); $list.each( function() { if ( $(this).text().toLowerCase().indexOf(search) === -1 ) $(this).hide(); else $(this).show(); } ); console.log('Time: ' + ((new Date).getTime() - jqStart)); } ); However, there was a couple of seconds delay after typing each letter (particularly the first letter). So I thought it may be slightly quicker if I used plain Javascript (I read recently that jQuery's each function is particularly slow). Here's my JS equivalent: document.getElementById('filter').addEventListener( 'keyup', function () { var jsStart = (new Date).getTime()...
3 comments
Read more

Is it possible to have IF statement in an Echo statement in PHP

By User
Thanks in advance. I did look at the other questions/answers that were similar and didn't find exactly what I was looking for. I'm trying to do this, am I on the right path? echo " <div id='tabs-".$match."'> <textarea id='".$match."' name='".$match."'>". if ($COLUMN_NAME === $match) { echo $FIELD_WITH_COLUMN_NAME; } else { } ."</textarea> <script type='text/javascript'> CKEDITOR.replace( '".$match."' ); </script> </div>"; I am getting the following error message in the browser: Parse error: syntax error, unexpected T_IF Please let me know if this is the right way to go about nesting an IF statement inside an echo. Thank you.
5 comments
Read more