Lately I've been switching to PHP from Perl, not just for server processing for basic webpages but for command line maintenance tasks as well. The syntax is so much less wonky, the core documentation solid, and the baked-in utility functions are so well thought out, that I haven't been missing Perl much at all. (On the other hand, I seem to be memorizing very few php functions names - but the process to get stuff done is still so fast, even with all the googling involved.)
(And like I said last year, PHP is much better than Javascript at dealing with global variables in functions.)
I've been running the user-submitted poetry site loveblender.com for decades, literally (if not so literarily). It's had a few overhauls, but it's got is still that bad old Perl, and I guess most of it isn't super-UTF8/16 aware. Usually that's not a problem, but sometimes (I think when someone copies and pastes from Word or what not) funky characters show up - smart quotes, ellipses, etc. I wanted to do a search and replace across my file store. (Guilty confession: for medium-size sites that don't need a lot cross-content grepping, I tend to turn first to flat files in folders- often many small ones, so something like atomicity is preserved even if things go wrong. It has some drawbacks, but I never have to worry about database conversions, and I obviously have a number of tools for hacking the content directly)
Anyway, here is the hack I came up with... and it had to do a lot of work, the Blender has had over 27,000 submissions over the years.
the core idea was to come up with a set of "replacers", and then do a simple search and replace with fixFile()... but to get there I need a few iterations of diagnostic work to figure out what the replacements should be, you can see lookForFunkyCharactersInFile() for oldschool "8-bit ASCII characters 128-255", mostly, and lookForFunkyEntityStringsInFile() for this longer bit of escaping that has been showing up lately. (For the former http://www.ascii-code.com/ was a huge help at figuring out what bad charaters were meant to be, I made a quick macro to import a big swath of the the foriegn characters in a single gulp.)
<?php
$directory = '/home/kirkjerk/domains/loveblender.com/blend/works';
$breaker = 0; # either only show one "funky character" type or 0 for all
$replacers = array(
"’" => "'",
"“" => "\"",
"”" => "\"",
"…" => "...",
"…" => "...",
"’" => "'",
"“" => "\"",
"”" => "\"",
chr(128) => "",
chr(131) => "ƒ",
chr(133) => "...",
chr(144) => "",
chr(145) => "'",
chr(146) => "'",
chr(147) => "\"",
chr(148) => "\"",
chr(150) => "-",
chr(151) => "--",
chr(152) => "", #probably some kind of soft hyphen
chr(153) => "", #probably some kind of soft hyphen
chr(156) => "", #probably some kind of soft hyphen
chr(157) => "", #probably some kind of soft hyphen
chr(160) => " ",
chr(161) => "¡",
chr(163) => "£",
chr(165) => "¥",
chr(166) => "",#probably some kind of soft hyphen
chr(167) => "§",
chr(169) => "(c)",
chr(170) => "",
chr(173) => "",
chr(174) => "(r)",
chr(180) => "'",
chr(183) => "*",
chr(189) => "1/2",
chr(226) => "'", #this one shows up even tho it's also â
chr(191) => "¿",chr(192) => "À",chr(193) => "Á",chr(194) => "Â",chr(195) => "Ã",
chr(196) => "Ä",chr(197) => "Å",chr(198) => "Æ",chr(199) => "Ç",chr(200) => "È",
chr(201) => "É",chr(202) => "Ê",chr(203) => "Ë",chr(204) => "Ì",chr(205) => "Í",
chr(206) => "Î",chr(207) => "Ï",chr(208) => "Ð",chr(209) => "Ñ",chr(210) => "Ò",
chr(211) => "Ó",chr(212) => "Ô",chr(213) => "Õ",chr(214) => "Ö",chr(215) => "×",
chr(216) => "Ø",chr(217) => "Ù",chr(218) => "Ú",chr(219) => "Û",chr(220) => "Ü",
chr(221) => "Ý",chr(222) => "Þ",chr(223) => "ß",chr(224) => "à",chr(225) => "á",
chr(227) => "ã",chr(228) => "ä",chr(229) => "å",chr(230) => "æ",
chr(231) => "ç",chr(232) => "è",chr(233) => "é",chr(234) => "ê",chr(235) => "ë",
chr(236) => "ì",chr(237) => "í",chr(238) => "î",chr(239) => "ï",chr(240) => "ð",
chr(241) => "ñ",chr(242) => "ò",chr(243) => "ó",chr(244) => "ô",chr(245) => "õ",
chr(246) => "ö",chr(247) => "÷",chr(248) => "ø",chr(249) => "ù",chr(250) => "ú",
chr(251) => "û",chr(252) => "ü",chr(253) => "ý",chr(254) => "þ",chr(255) => "ÿ"
);
$it = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($directory));
$it->rewind();
while($it->valid()) {
$pathname = $it->getSubPathName();
if (endsWith($pathname, ".work")) {
#lookForFunkyCharactersInFile($pathname);
#lookForFunkyEntityStringsInFile($pathname);
fixFile($pathname);
}
$it->next();
}
function fixFile($pathname){
$guts = file_get_contents($pathname);
$guts = replaceBads($guts);
file_put_contents($pathname,$guts);
}
function lookForFunkyEntityStringsInFile($pathname){
$lines = file($pathname);
foreach($lines as $line){
$line = replaceBads($line);
if(strpos($line,"â")){
$url = pathnameToLink($pathname);
print $pathname." ".$url."\n".$line."\n";
}
}
}
function lookForFunkyCharactersInFile($pathname){
global $breaker;
$lines = file($pathname);
foreach($lines as $line){
$line = replaceBads($line);
$lineIfFunky = isFunky($line,$breaker);
if($lineIfFunky != "") {
$url = pathnameToLink($pathname);
print $pathname." ".$url."\n".$line.$lineIfFunky."\n";
}
}
}
function isFunky($line,$ugh){
for($i = 0; $i < strlen($line); $i++){
$c = substr($line,$i,1);
if(
($ugh == 0 && ord($c) >= 128) ||
($ugh != 0 && ord($c) == $ugh)
) {
return str_repeat(" ",$i)."^ ------ at pos $i got ".ord($c);
}
}
return "";
}
function pathnameToLink($path){
return "http://www.loveblender.com/blend/wv.cgi?id=".str_replace(".work","",str_replace("/",".",$path));
}
function replaceBads($line){
global $replacers;
foreach ($replacers as $badFrom => $badTo){
$pos = strpos($line,$badFrom);
if ($pos !== false) {
# print " FROM ".$line;
$line = str_replace($badFrom,$badTo,$line);
# print " TO ".$line;
}
}
return $line;
}
function endsWith($haystack, $needle) {
return $needle === "" || (($temp = strlen($haystack) - strlen($needle)) >= 0 && strpos($haystack, $needle, $temp) !== false);
}
?>
No comments:
Post a Comment