Browsing the web from the command line

[mala]

$ cat index.htm | sed 's/<[^>]*>//g'

... and it would remove all the tags from the input file (here: index.htm) under certain circumstances (to be explained by mala !!!).

Hmm... I don't know if there's anything more (I'm not good with sed) but AT LEAST the tag should be all on one line: this is because the input file isn't considered as one line, thing that with Perl could be done with /gs instead of /g (g=global, s=single line). Hey, what about sed?

... hope that's all

[mala]

$ lynx -source http://ret.mine.nu/top.swf | strings | grep http

People, this oneliner really made me think: it wasn't just the start of this _beautiful_ thread, but left me with the desire of making flash scripts accessible -at least for what concerns links- to flash-unabled users. That oneliner has a problem: it catches only the urls which start with http, that is only absolute ones. Reversing some flash files and applying the right regexp I created this little -but, I hope, useful- script:

#!/usr/bin/perl



undef $/;

$_ = <>;



# SYNTAX IS: 0x00 0x83 0xlength 0x00 "string" 0x00

while (/x00x83.x00(.*?)x00/gs){

 * * * *print "$1n";

}

That's all. Call it however you like (let's say flash.pl) and then:

$ lynx -dump http://ret.mine.nu/top.swf | perl flash.pl
or
$ lynx -source http://ret.mine.nu/top.swf | perl flash.pl

will give you:

/
members.html
ret.viewer?p=Essays
ret.viewer?p=Tools
ret.viewer?p=Challenges
ret.viewer?p=Console
ret.viewer?p=Stegano
http://ret.didjitalyphrozen.com/board/index.php
links.html
contact.html

[null]

Hmm... I don't know if there's anything more (I'm not good with sed) but AT LEAST the tag should be all on one line: this is because the input file isn't considered as one line, thing that with Perl could be done with /gs instead of /g (g=global, s=single line). Hey, what about sed? *

Yes, this is what I meant - every tag should be on a single line. Thanks for the explanation. I don't think that sed offers any options to transform the input into a single line, but ...

... we are using the UNIX shell, so never be affraid of a new challenge! You can solve _almost everything_ from the shell. Here, tr is our friend:

[b]$ cat index.htm | tr -d '

[scorer]

mala, in your exturl.pl script, you stated it doesn't convert relative URLs to absolute ones.
i found an easy workaround, that you may already know, for this.

use URI:URL;

sub rel2abs {
my $rel = shift;
my $base = shift;
$uri = URI->new_abs($rel, $base);
return $uri->as_string;
}

for example:
$rel = /file.zip
$base = http://server.org/path/index.html
outputs => http://server.org/file.zip
and
$rel = file.zip
$base = http://server.org/path/index.html
outputs => http://server.org/path/file.zip

you may want to play with it to see how it behaves.

~

aside from this, i'm currently working out a searching script based on ideas found in essays on searchlores (lexi_wot, lexi_lau).

i have some (simple) ideas about how to improve on this, and would enjoy to share these, and some codes.
let me know if you're on the same track.
(the goal here is to build something able to parse results from *any* SE/database/searching facility...)

[mala]

Hi!

I'm sorry I've been away for so much time... I had some projects going on which took quite much time. But now I've written a nice crypto paper for beginners (unfortunately, it's encrypted itself... with an algorithm called Italian: anyone who wants to read it anyway, and maybe translate it, is welcome :wink and made some experiments with genetic algorithms (I've done my first genetic keygen! :wink

mala, in your exturl.pl script, you stated it doesn't convert relative URLs to absolute ones.
i found an easy workaround, that you may already know, for this.

Thank you very much, in the meanwhile I already had updated that script, because I needed one which gave me absolute urls. The state of the work now is that I can connect to any PhpBB messageboard and dump all its messages on my hd... it's awful (one big HTML file at the moment) but I'm working on the mysql interface in these days

aside from this, i'm currently working out a searching script based on ideas found in essays on searchlores (lexi_wot, lexi_lau).

i have some (simple) ideas about how to improve on this, and would enjoy to share these, and some codes.
let me know if you're on the same track.
(the goal here is to build something able to parse results from *any* SE/database/searching facility...)

It's a very nice topic and it's quite similar to the one I'd like to carry on: accessing the Web in an intelligent way, keeping the interesting content and stripping away the uninteresting one. So please, if you like and have the time share your ideas and I'm sure many will participate (but I might be wrong ).

[mala]

Hi ppl!

I know much time passed, but I've worked a little on that flesh regexp this afternoon and thought this might be useful to someone: given an url, the script downloads a flash file and extracts its urls; then it looks ahead, in the linked pages, the <title> and creates some html code which contains a "usable" menu.
I think my next step will be to join this one to a little proxy (I have some ready source code for this) and make an automatic converter which makes flash websites accessible to browsers which are not flash capable... will let you know

(hey, thank you very much again, scorer, for your url conversion sub: as you can see, I've used it here! )

#!/usr/bin/perl



use LWP::Simple;

use URI::URL;



sub rel2abs {

 * * * *my ($rel,$base) = @_;

 * * * *my $uri = URI->new_abs($rel, $base);

 * * * *return $uri->as_string;

}



$url * = $ARGV[0];

$flash = get($url) || die "Could not download $ARGV[0]";



# SYNTAX IS: 0x00 0x83 0xlength 0x00 "string" 0x00

while ($flash =~ /x00x83.x00(.*?)x00/gs){

 * * * *my $nextitle;

 * * * *my $link = rel2abs ($1,$url);

 * * * *my $nextpage = get ($link);

 * * * *if ($nextpage =~ /<title>(.*?)</title>/i){

 * * * * * * * *$nextitle = $1;

 * * * *}else{

 * * * * * * * *$nextitle = $link;

 * * * *}

 * * * *print qq|<a href="$link">$nextitle</a><br>n|;

}

[mala]

... I've just uploaded the first release of TWO on sourceforge:

http://two.sf.net

I hope this will help, if not in browsing forums offline effectively, at least in creating new power browsing tools. Plese, let me know what you think of it and if you found useful for your purposes. Any feedback is appreciated (insults only by private mail )